3 May 2024 | Margarida M. Campos, António Farinhas, Chrysoula Zerva, Mário A.T. Figueiredo, André F.T. Martins
This paper provides a comprehensive survey of conformal prediction (CP) techniques and their applications in Natural Language Processing (NLP). CP is a theoretically sound and practically useful framework that combines flexibility with strong statistical guarantees, making it particularly promising for addressing the current shortcomings of NLP systems, such as hallucinations, poor calibration, and lack of uncertainty quantification. The survey covers the theory and guarantees of CP, its variants, and its applications in various NLP tasks, including text classification, sequence tagging, natural language generation, and uncertainty-based evaluation. It also discusses future research directions and open challenges, such as handling label variation, fairness, and data limitations. The paper highlights the potential of CP to improve the reliability and robustness of NLP systems, particularly in critical applications.This paper provides a comprehensive survey of conformal prediction (CP) techniques and their applications in Natural Language Processing (NLP). CP is a theoretically sound and practically useful framework that combines flexibility with strong statistical guarantees, making it particularly promising for addressing the current shortcomings of NLP systems, such as hallucinations, poor calibration, and lack of uncertainty quantification. The survey covers the theory and guarantees of CP, its variants, and its applications in various NLP tasks, including text classification, sequence tagging, natural language generation, and uncertainty-based evaluation. It also discusses future research directions and open challenges, such as handling label variation, fairness, and data limitations. The paper highlights the potential of CP to improve the reliability and robustness of NLP systems, particularly in critical applications.