The paper "A Decision Theoretic Framework for Measuring AI Reliance" by Ziyang Guo, Yifan Wu, Jason Hartline, and Jessica Hullman from Northwestern University proposes a formal definition of reliance in AI-advised decision-making. The authors argue that the current definition of appropriate reliance, which is based on the probability of following an AI recommendation when it is correct, lacks statistical grounding and can lead to contradictions. They introduce a new framework grounded in statistical decision theory that separates the concepts of reliance and the challenges a human may face in differentiating signals and forming accurate beliefs. The framework defines reliance as the conditional probability that a decision-maker chooses the AI recommendation when the AI's recommendation differs from the human's. It also introduces the rational decision-maker, who perfectly perceives the provided information and chooses the optimal action, and the behavioral decision-maker, who makes decisions based on their own predictions and the AI's recommendations. The framework includes a rational baseline and a rational benchmark to measure the maximum and minimum attainable performance, respectively. The authors apply their framework to three well-regarded AI-advised decision-making studies, demonstrating how it can separate the loss due to mis-reliance from the loss due to not accurately differentiating signals. They show that the primary source of performance loss in these studies is not over-reliance on the AI but rather the inability to accurately distinguish when the AI is better than the human. This finding suggests that improving the human's ability to differentiate between the AI and human recommendations is more critical than calibrating their reliance on the AI. The paper concludes by emphasizing the importance of decoupling sources of behavioral loss in human-AI team performance and providing clear comparison points for interpreting study results. This framework helps researchers identify the upper bound of complementary performance and how far the human-AI team is from optimal performance.The paper "A Decision Theoretic Framework for Measuring AI Reliance" by Ziyang Guo, Yifan Wu, Jason Hartline, and Jessica Hullman from Northwestern University proposes a formal definition of reliance in AI-advised decision-making. The authors argue that the current definition of appropriate reliance, which is based on the probability of following an AI recommendation when it is correct, lacks statistical grounding and can lead to contradictions. They introduce a new framework grounded in statistical decision theory that separates the concepts of reliance and the challenges a human may face in differentiating signals and forming accurate beliefs. The framework defines reliance as the conditional probability that a decision-maker chooses the AI recommendation when the AI's recommendation differs from the human's. It also introduces the rational decision-maker, who perfectly perceives the provided information and chooses the optimal action, and the behavioral decision-maker, who makes decisions based on their own predictions and the AI's recommendations. The framework includes a rational baseline and a rational benchmark to measure the maximum and minimum attainable performance, respectively. The authors apply their framework to three well-regarded AI-advised decision-making studies, demonstrating how it can separate the loss due to mis-reliance from the loss due to not accurately differentiating signals. They show that the primary source of performance loss in these studies is not over-reliance on the AI but rather the inability to accurately distinguish when the AI is better than the human. This finding suggests that improving the human's ability to differentiate between the AI and human recommendations is more critical than calibrating their reliance on the AI. The paper concludes by emphasizing the importance of decoupling sources of behavioral loss in human-AI team performance and providing clear comparison points for interpreting study results. This framework helps researchers identify the upper bound of complementary performance and how far the human-AI team is from optimal performance.