The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels

The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels

9 May 2024 | Eve Fleisig, Su Lin Blodgett, Dan Klein, Zeerak Talat
This paper explores the paradigm shift from the traditional data labeling paradigm to the *perspectivist* approach, which views annotator disagreement as a valuable source of information rather than a problem to minimize. The authors examine the assumptions and challenges surrounding the causes of disagreement, both in the traditional and perspectivist paradigms. They highlight that the traditional paradigm often attributes disagreement to biased or inept annotators, while the perspectivist approach recognizes that demographic and lived experience factors can also contribute to disagreement. The paper discusses practical and normative challenges, such as data quality, evaluation metrics, and ethical considerations, and provides recommendations for designing more inclusive and accurate data labeling processes. The authors argue that by explicitly addressing these challenges, the perspectivist approach can better capture the full spectrum of human perspectives and improve the performance and calibration of machine learning models.This paper explores the paradigm shift from the traditional data labeling paradigm to the *perspectivist* approach, which views annotator disagreement as a valuable source of information rather than a problem to minimize. The authors examine the assumptions and challenges surrounding the causes of disagreement, both in the traditional and perspectivist paradigms. They highlight that the traditional paradigm often attributes disagreement to biased or inept annotators, while the perspectivist approach recognizes that demographic and lived experience factors can also contribute to disagreement. The paper discusses practical and normative challenges, such as data quality, evaluation metrics, and ethical considerations, and provides recommendations for designing more inclusive and accurate data labeling processes. The authors argue that by explicitly addressing these challenges, the perspectivist approach can better capture the full spectrum of human perspectives and improve the performance and calibration of machine learning models.
Reach us at info@study.space