The impact of AI errors in a human-in-the-loop process

The impact of AI errors in a human-in-the-loop process

2024 | Ujué Agudo¹², Karlos G. Liberal¹, Miren Arrese¹ and Helena Matute²*
This study investigates the impact of AI errors in human-in-the-loop decision-making processes. Automated decision-making is increasingly used in the public sector, and human oversight is recommended to prevent biased or erroneous algorithmic decisions. However, the scientific literature on human-in-the-loop performance is inconclusive about the benefits and risks of human presence in such processes. Two experiments were conducted to simulate an automated decision-making process where participants judged defendants based on various crimes, and the timing of AI support (before or after their judgments) was manipulated. The results showed that human judgment was affected when incorrect AI support was provided, particularly when it was given before the participants made their own judgment, leading to reduced accuracy. The data and materials are available on the Open Science Framework. The presence of AI in public sector decisions, such as in justice, health, and education, is becoming more common. Automated decision support systems are used in many countries, including the US, UK, China, Estonia, Argentina, Poland, and Spain. These systems often support human decision-making by providing information or recommendations, but they do not make decisions autonomously. This human-in-the-loop process aims to ensure better decisions through human supervision and intervention. However, there is a tendency for humans to comply with AI recommendations, known as automation bias, which can lead to errors. Empirical evidence suggests that humans may not always override AI decisions, and automation bias is well-documented in fields like aviation, healthcare, and military. In the judicial system, recent studies show less consistent effects of automation bias. For example, the Spanish system RisCanvi has a low predictive accuracy but is widely used, with government officials rarely disagreeing with its recommendations. However, other studies suggest that participants may adjust their judgments when given AI support, indicating lower automation bias. The study's experiments aimed to test whether manipulating the timing of AI support in human-in-the-loop processes could improve decision accuracy and reduce automation bias. In Experiment 1, participants were asked to judge defendants' guilt, with AI support either before or after their judgment. The results showed that participants who judged before receiving AI support were more accurate, especially when the AI was incorrect. In Experiment 2, a larger sample and a 0-100 scale were used, and similar results were found, with participants who judged before AI support being more accurate in incorrect cases. The findings suggest that forcing participants to emit their judgment before receiving AI support can reduce automation bias and improve decision accuracy. This approach could help mitigate the risks associated with AI errors in human-in-the-loop processes.This study investigates the impact of AI errors in human-in-the-loop decision-making processes. Automated decision-making is increasingly used in the public sector, and human oversight is recommended to prevent biased or erroneous algorithmic decisions. However, the scientific literature on human-in-the-loop performance is inconclusive about the benefits and risks of human presence in such processes. Two experiments were conducted to simulate an automated decision-making process where participants judged defendants based on various crimes, and the timing of AI support (before or after their judgments) was manipulated. The results showed that human judgment was affected when incorrect AI support was provided, particularly when it was given before the participants made their own judgment, leading to reduced accuracy. The data and materials are available on the Open Science Framework. The presence of AI in public sector decisions, such as in justice, health, and education, is becoming more common. Automated decision support systems are used in many countries, including the US, UK, China, Estonia, Argentina, Poland, and Spain. These systems often support human decision-making by providing information or recommendations, but they do not make decisions autonomously. This human-in-the-loop process aims to ensure better decisions through human supervision and intervention. However, there is a tendency for humans to comply with AI recommendations, known as automation bias, which can lead to errors. Empirical evidence suggests that humans may not always override AI decisions, and automation bias is well-documented in fields like aviation, healthcare, and military. In the judicial system, recent studies show less consistent effects of automation bias. For example, the Spanish system RisCanvi has a low predictive accuracy but is widely used, with government officials rarely disagreeing with its recommendations. However, other studies suggest that participants may adjust their judgments when given AI support, indicating lower automation bias. The study's experiments aimed to test whether manipulating the timing of AI support in human-in-the-loop processes could improve decision accuracy and reduce automation bias. In Experiment 1, participants were asked to judge defendants' guilt, with AI support either before or after their judgment. The results showed that participants who judged before receiving AI support were more accurate, especially when the AI was incorrect. In Experiment 2, a larger sample and a 0-100 scale were used, and similar results were found, with participants who judged before AI support being more accurate in incorrect cases. The findings suggest that forcing participants to emit their judgment before receiving AI support can reduce automation bias and improve decision accuracy. This approach could help mitigate the risks associated with AI errors in human-in-the-loop processes.
Reach us at info@study.space
[slides] The impact of AI errors in a human-in-the-loop process | StudySpace