Fairness in Criminal Justice Risk Assessments: The State of the Art

Fairness in Criminal Justice Risk Assessments: The State of the Art

May 30, 2017 | Richard Berka, Hoda Heidari, Shahin Jabbari, Michael Kearns, Aaron Roth
The paper discusses fairness in criminal justice risk assessments, highlighting the trade-offs between fairness and accuracy. It clarifies that there are at least six types of fairness, some of which are incompatible with each other and with accuracy. The authors argue that it is generally impossible to maximize both accuracy and fairness simultaneously, and that different fairness criteria may conflict. The paper also emphasizes the role of base rates across protected groups, which can complicate fairness assessments. The authors examine various definitions of fairness, including overall accuracy equality, statistical parity, conditional procedure accuracy equality, conditional use accuracy equality, treatment equality, and total fairness. They show that these definitions are related and that achieving one often requires sacrificing another. The paper also discusses the challenges of estimating accuracy and fairness, noting that even asymptotically, estimates may be biased. The authors present examples of confusion tables to illustrate how different fairness criteria can be applied and how they may conflict. They also discuss the impossibility of achieving certain fairness goals, such as conditional use accuracy equality and equal false positive and false negative rates, unless base rates are equal or there is perfect separation. The paper concludes that while there are various proposed solutions to improve fairness, achieving a balance between fairness and accuracy remains challenging. The authors emphasize the need for careful consideration of trade-offs and the importance of understanding the underlying data and population characteristics.The paper discusses fairness in criminal justice risk assessments, highlighting the trade-offs between fairness and accuracy. It clarifies that there are at least six types of fairness, some of which are incompatible with each other and with accuracy. The authors argue that it is generally impossible to maximize both accuracy and fairness simultaneously, and that different fairness criteria may conflict. The paper also emphasizes the role of base rates across protected groups, which can complicate fairness assessments. The authors examine various definitions of fairness, including overall accuracy equality, statistical parity, conditional procedure accuracy equality, conditional use accuracy equality, treatment equality, and total fairness. They show that these definitions are related and that achieving one often requires sacrificing another. The paper also discusses the challenges of estimating accuracy and fairness, noting that even asymptotically, estimates may be biased. The authors present examples of confusion tables to illustrate how different fairness criteria can be applied and how they may conflict. They also discuss the impossibility of achieving certain fairness goals, such as conditional use accuracy equality and equal false positive and false negative rates, unless base rates are equal or there is perfect separation. The paper concludes that while there are various proposed solutions to improve fairness, achieving a balance between fairness and accuracy remains challenging. The authors emphasize the need for careful consideration of trade-offs and the importance of understanding the underlying data and population characteristics.
Reach us at info@study.space