AutoEval Done Right: Using Synthetic Data for Model Evaluation This paper introduces a method for evaluating machine learning models using synthetic data generated by AI, which reduces the need for human-labeled data. The proposed approach, called autoevaluation, involves generating synthetic labels using AI on a large unlabeled dataset and then evaluating AI models using these synthetic labels. This method improves sample efficiency while maintaining statistical validity and reduces the effective human-labeled sample size by up to 50% in experiments with GPT-4. The paper presents a statistical framework for autoevaluation that uses prediction-powered inference (PPI) to estimate model performance. PPI is a set of estimators that incorporate predictions from machine learning models to produce lower-variance, unbiased estimates. The authors describe an optimized variant of PPI, PPI++, which is used to estimate metrics using synthetic data. PPI++ is shown to provide tighter confidence intervals and more accurate point estimates compared to classical methods. The paper applies the proposed method to evaluate computer vision models and protein fitness prediction models. In the computer vision experiments, PPI++ significantly improves the mean-squared error of model accuracy estimates and increases the effective sample size compared to classical methods. In the protein fitness experiments, PPI++ provides more accurate rankings of models and tighter confidence intervals for model correlations with experimental fitness scores. The paper also discusses the application of autoevaluation to evaluate models based on pairwise comparisons. It introduces the Bradley-Terry (BT) model for ranking models based on pairwise comparisons and applies PPI++ to estimate BT coefficients. The results show that PPI++ provides more accurate and reliable estimates of model rankings compared to classical methods. The authors conclude that autoevaluation using synthetic data can significantly reduce the cost and effort of model evaluation while maintaining statistical validity. The proposed method is applicable to a wide range of tasks and can be implemented using existing Python software. The paper also discusses the broader implications of autoevaluation, including its potential to improve algorithmic fairness and enable more efficient human oversight of model evaluation.AutoEval Done Right: Using Synthetic Data for Model Evaluation This paper introduces a method for evaluating machine learning models using synthetic data generated by AI, which reduces the need for human-labeled data. The proposed approach, called autoevaluation, involves generating synthetic labels using AI on a large unlabeled dataset and then evaluating AI models using these synthetic labels. This method improves sample efficiency while maintaining statistical validity and reduces the effective human-labeled sample size by up to 50% in experiments with GPT-4. The paper presents a statistical framework for autoevaluation that uses prediction-powered inference (PPI) to estimate model performance. PPI is a set of estimators that incorporate predictions from machine learning models to produce lower-variance, unbiased estimates. The authors describe an optimized variant of PPI, PPI++, which is used to estimate metrics using synthetic data. PPI++ is shown to provide tighter confidence intervals and more accurate point estimates compared to classical methods. The paper applies the proposed method to evaluate computer vision models and protein fitness prediction models. In the computer vision experiments, PPI++ significantly improves the mean-squared error of model accuracy estimates and increases the effective sample size compared to classical methods. In the protein fitness experiments, PPI++ provides more accurate rankings of models and tighter confidence intervals for model correlations with experimental fitness scores. The paper also discusses the application of autoevaluation to evaluate models based on pairwise comparisons. It introduces the Bradley-Terry (BT) model for ranking models based on pairwise comparisons and applies PPI++ to estimate BT coefficients. The results show that PPI++ provides more accurate and reliable estimates of model rankings compared to classical methods. The authors conclude that autoevaluation using synthetic data can significantly reduce the cost and effort of model evaluation while maintaining statistical validity. The proposed method is applicable to a wide range of tasks and can be implemented using existing Python software. The paper also discusses the broader implications of autoevaluation, including its potential to improve algorithmic fairness and enable more efficient human oversight of model evaluation.