This paper evaluates the real-world applicability of semi-supervised learning (SSL) algorithms by addressing limitations in current benchmarking practices. The authors argue that existing benchmarks fail to reflect the challenges SSL faces in real-world scenarios, such as class distribution mismatches, limited labeled data, and the presence of out-of-distribution examples in unlabeled data. To address these issues, they present a unified reimplementation of various SSL techniques and evaluate them on a suite of experiments designed to simulate real-world conditions. The study finds that simple baselines without unlabeled data often perform better than reported, SSL methods vary in their sensitivity to the amount of labeled and unlabeled data, and performance can degrade significantly when unlabeled data contains different class distributions than labeled data. The authors also highlight the importance of using the same underlying model for evaluating different SSL algorithms to avoid conflating comparisons. Key findings include: when given equal hyperparameter tuning budgets, the performance gap between SSL and fully supervised methods is smaller than typically reported; large classifiers with careful regularization can achieve high accuracy with minimal labeled data; pre-training on a different dataset and retraining on the target dataset can outperform many SSL methods; SSL performance can degrade drastically when unlabeled data has a different class distribution than labeled data; and different SSL methods exhibit varying levels of sensitivity to the amount of labeled and unlabeled data. The authors also emphasize the importance of using realistically small validation sets for evaluating SSL methods, as current benchmarks often use validation sets significantly larger than the training set, leading to unreliable comparisons. They propose a new experimental methodology that includes a unified reimplementation of SSL techniques and a standardized evaluation framework. The paper concludes with recommendations for evaluating SSL algorithms, including using the same underlying model for comparisons, reporting well-tuned fully supervised and transfer learning baselines, varying the amount of labeled and unlabeled data, and being cautious about over-tuning hyperparameters on large validation sets. The authors argue that SSL is most appropriate when there are no high-quality labeled datasets from similar domains, when labeled data is collected from the same distribution as unlabeled data, and when the labeled dataset is large enough to accurately estimate validation accuracy. The unified implementation and evaluation platform are made publicly available to help guide SSL research towards real-world applicability.This paper evaluates the real-world applicability of semi-supervised learning (SSL) algorithms by addressing limitations in current benchmarking practices. The authors argue that existing benchmarks fail to reflect the challenges SSL faces in real-world scenarios, such as class distribution mismatches, limited labeled data, and the presence of out-of-distribution examples in unlabeled data. To address these issues, they present a unified reimplementation of various SSL techniques and evaluate them on a suite of experiments designed to simulate real-world conditions. The study finds that simple baselines without unlabeled data often perform better than reported, SSL methods vary in their sensitivity to the amount of labeled and unlabeled data, and performance can degrade significantly when unlabeled data contains different class distributions than labeled data. The authors also highlight the importance of using the same underlying model for evaluating different SSL algorithms to avoid conflating comparisons. Key findings include: when given equal hyperparameter tuning budgets, the performance gap between SSL and fully supervised methods is smaller than typically reported; large classifiers with careful regularization can achieve high accuracy with minimal labeled data; pre-training on a different dataset and retraining on the target dataset can outperform many SSL methods; SSL performance can degrade drastically when unlabeled data has a different class distribution than labeled data; and different SSL methods exhibit varying levels of sensitivity to the amount of labeled and unlabeled data. The authors also emphasize the importance of using realistically small validation sets for evaluating SSL methods, as current benchmarks often use validation sets significantly larger than the training set, leading to unreliable comparisons. They propose a new experimental methodology that includes a unified reimplementation of SSL techniques and a standardized evaluation framework. The paper concludes with recommendations for evaluating SSL algorithms, including using the same underlying model for comparisons, reporting well-tuned fully supervised and transfer learning baselines, varying the amount of labeled and unlabeled data, and being cautious about over-tuning hyperparameters on large validation sets. The authors argue that SSL is most appropriate when there are no high-quality labeled datasets from similar domains, when labeled data is collected from the same distribution as unlabeled data, and when the labeled dataset is large enough to accurately estimate validation accuracy. The unified implementation and evaluation platform are made publicly available to help guide SSL research towards real-world applicability.