Obtaining Well Calibrated Probabilities Using Bayesian Binning

Obtaining Well Calibrated Probabilities Using Bayesian Binning

2015 | Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht
This paper introduces a new non-parametric calibration method called Bayesian Binning into Quantiles (BBQ) for improving the calibration of binary classifiers. The method is a post-processing technique that enhances the calibration of existing classification algorithms without modifying their learning process. BBQ addresses the limitations of existing calibration methods such as histogram binning, Platt scaling, and isotonic regression by considering multiple binning models and their combinations. It uses a Bayesian score derived from the BDeu score to select and combine binning models, leading to more accurate and robust calibrated predictions. BBQ works by partitioning the output space into bins and using the distribution of true labels within each bin to estimate calibrated probabilities. The method considers multiple binning models with different numbers of bins and combines them using a Bayesian score to improve calibration. The calibration performance is evaluated using metrics such as Expected Calibration Error (ECE) and Maximum Calibration Error (MCE), which measure how well the predicted probabilities match the actual outcomes. Experiments on both simulated and real datasets show that BBQ outperforms existing calibration methods in terms of calibration performance. On simulated data, BBQ performs better than Platt's method and isotonic regression, especially when using a linear SVM as the base learner. On real data, BBQ is statistically superior to other calibration methods in terms of ECE and MCE, and performs as well as isotonic regression in terms of discrimination. The results indicate that BBQ is a competitive and effective method for improving the calibration of binary classifiers. Future work includes investigating the theoretical properties of BBQ and extending it to multi-class and multi-label calibration problems.This paper introduces a new non-parametric calibration method called Bayesian Binning into Quantiles (BBQ) for improving the calibration of binary classifiers. The method is a post-processing technique that enhances the calibration of existing classification algorithms without modifying their learning process. BBQ addresses the limitations of existing calibration methods such as histogram binning, Platt scaling, and isotonic regression by considering multiple binning models and their combinations. It uses a Bayesian score derived from the BDeu score to select and combine binning models, leading to more accurate and robust calibrated predictions. BBQ works by partitioning the output space into bins and using the distribution of true labels within each bin to estimate calibrated probabilities. The method considers multiple binning models with different numbers of bins and combines them using a Bayesian score to improve calibration. The calibration performance is evaluated using metrics such as Expected Calibration Error (ECE) and Maximum Calibration Error (MCE), which measure how well the predicted probabilities match the actual outcomes. Experiments on both simulated and real datasets show that BBQ outperforms existing calibration methods in terms of calibration performance. On simulated data, BBQ performs better than Platt's method and isotonic regression, especially when using a linear SVM as the base learner. On real data, BBQ is statistically superior to other calibration methods in terms of ECE and MCE, and performs as well as isotonic regression in terms of discrimination. The results indicate that BBQ is a competitive and effective method for improving the calibration of binary classifiers. Future work includes investigating the theoretical properties of BBQ and extending it to multi-class and multi-label calibration problems.
Reach us at info@study.space