25 May 2024 | Hunter Lang, David Sontag, Aravindan Vijayaraghavan
The paper discusses the phenomenon of weak-to-strong generalization in weakly-supervised learning, where a strong model trained on weak labels can correct the errors of a weaker teacher model and perform well on examples not covered by the weak labels. The existing theoretical literature on weak supervision often fails to account for this effect, which the authors refer to as *pseudolabel correction* and *coverage expansion*. They introduce a new bound based on the *expansion* properties of the data distribution and the student hypothesis class, which directly captures these effects. The bound is derived using an expansion condition that ensures that "bad" points (incorrectly labeled or unlabelled) have many "good" neighbors (correctly labeled points). This condition allows the authors to prove a relationship between the student model's error on the weak labels and its error on the true labels. The paper also provides a statistical method to check the expansion conditions from finite data and empirical evidence that these conditions hold in practice. The results are the first error bounds for programmatic weak supervision with realistic assumptions and generalize several existing results from co-training, self-training, and distribution shift literature. The authors also discuss the limitations and future directions of their work.The paper discusses the phenomenon of weak-to-strong generalization in weakly-supervised learning, where a strong model trained on weak labels can correct the errors of a weaker teacher model and perform well on examples not covered by the weak labels. The existing theoretical literature on weak supervision often fails to account for this effect, which the authors refer to as *pseudolabel correction* and *coverage expansion*. They introduce a new bound based on the *expansion* properties of the data distribution and the student hypothesis class, which directly captures these effects. The bound is derived using an expansion condition that ensures that "bad" points (incorrectly labeled or unlabelled) have many "good" neighbors (correctly labeled points). This condition allows the authors to prove a relationship between the student model's error on the weak labels and its error on the true labels. The paper also provides a statistical method to check the expansion conditions from finite data and empirical evidence that these conditions hold in practice. The results are the first error bounds for programmatic weak supervision with realistic assumptions and generalize several existing results from co-training, self-training, and distribution shift literature. The authors also discuss the limitations and future directions of their work.