Bias in Computer Systems

Bias in Computer Systems

July 1996 | BATYA FRIEDMAN and HELEN NISSENBAUM
Bias in computer systems arises in three categories: preexisting, technical, and emergent. Preexisting bias stems from social institutions, practices, and attitudes. Technical bias results from technical constraints or considerations. Emergent bias arises in the context of use. The authors argue that bias in computer systems should be considered a key criterion for judging system quality, alongside reliability, accuracy, and efficiency. The article discusses how bias can manifest in computer systems, using examples such as airline reservation systems and the National Resident Match Program (NRMP). In the case of airline reservation systems, algorithms favoring "on-line" flights can systematically disadvantage international carriers. In the NRMP, the algorithm's preference for hospital programs over students can be seen as preexisting bias, as it reflects institutional practices rather than technical constraints. The authors propose a framework for analyzing bias in computer systems, distinguishing between preexisting, technical, and emergent bias. Preexisting bias is rooted in social institutions and practices. Technical bias arises from technical constraints. Emergent bias emerges in the context of use, often due to changes in societal conditions. The article also discusses how bias can be minimized in system design. Designers must consider the social context and potential biases in the system's use. Techniques such as rapid prototyping, formative evaluation, and field testing can help identify and mitigate bias. Additionally, designers should anticipate probable contexts of use and account for diverse social contexts. The authors conclude that bias in computer systems should be addressed as a critical issue in system design. While some biases are deeply rooted in societal structures, others can be mitigated through thoughtful design and implementation. However, resolving biases in larger societal problems goes beyond system design and requires broader societal efforts.Bias in computer systems arises in three categories: preexisting, technical, and emergent. Preexisting bias stems from social institutions, practices, and attitudes. Technical bias results from technical constraints or considerations. Emergent bias arises in the context of use. The authors argue that bias in computer systems should be considered a key criterion for judging system quality, alongside reliability, accuracy, and efficiency. The article discusses how bias can manifest in computer systems, using examples such as airline reservation systems and the National Resident Match Program (NRMP). In the case of airline reservation systems, algorithms favoring "on-line" flights can systematically disadvantage international carriers. In the NRMP, the algorithm's preference for hospital programs over students can be seen as preexisting bias, as it reflects institutional practices rather than technical constraints. The authors propose a framework for analyzing bias in computer systems, distinguishing between preexisting, technical, and emergent bias. Preexisting bias is rooted in social institutions and practices. Technical bias arises from technical constraints. Emergent bias emerges in the context of use, often due to changes in societal conditions. The article also discusses how bias can be minimized in system design. Designers must consider the social context and potential biases in the system's use. Techniques such as rapid prototyping, formative evaluation, and field testing can help identify and mitigate bias. Additionally, designers should anticipate probable contexts of use and account for diverse social contexts. The authors conclude that bias in computer systems should be addressed as a critical issue in system design. While some biases are deeply rooted in societal structures, others can be mitigated through thoughtful design and implementation. However, resolving biases in larger societal problems goes beyond system design and requires broader societal efforts.
Reach us at info@study.space
[slides] Bias in computer systems | StudySpace