A Decision Theoretic Framework for Measuring AI Reliance Ziyang Guo, Yifan Wu, Jason Hartline, and Jessica Hullman propose a formal definition of AI reliance based on statistical decision theory to address the lack of formal grounding in current definitions of appropriate reliance in human-AI decision-making. They argue that the conventional binary definition of appropriate reliance leads to contradictions and misinterpretations of experimental results. Their framework separates reliance as the probability a human follows an AI recommendation from challenges in forming accurate beliefs about the situation. The framework defines a benchmark for complementary performance representing the maximum attainable performance with AI and human cooperation, and a baseline for performance without cooperation. They apply the framework to three AI-advised decision-making experiments, showing how it can separate loss due to mis-reliance from loss due to inaccurate signal differentiation. The framework evaluates these losses by comparing to a baseline and a benchmark for complementary performance defined by the expected payoff of a rational decision-maker. The framework allows for a continuous assessment of reliance in payoff space, enabling more fine-grained analysis. The authors define appropriate reliance as the rational decision-maker's reliance level on the AI, conditional on the AI recommendation being different from the human recommendation. They also define behavioral under-reliance and over-reliance by comparing the behavioral reliance level to the appropriate reliance level. The framework separates the loss into reliance loss and discrimination loss, where reliance loss is the loss from over- or under-reliance on the AI, and discrimination loss is the loss from not accurately differentiating when the AI is better than the human. The framework is applied to three experiments, demonstrating how it can better reveal the limits of human performance and specific sources of behavioral loss. The authors conclude that the framework provides a valuable tool for evaluating AI reliance and understanding the sources of performance loss in human-AI teams.A Decision Theoretic Framework for Measuring AI Reliance Ziyang Guo, Yifan Wu, Jason Hartline, and Jessica Hullman propose a formal definition of AI reliance based on statistical decision theory to address the lack of formal grounding in current definitions of appropriate reliance in human-AI decision-making. They argue that the conventional binary definition of appropriate reliance leads to contradictions and misinterpretations of experimental results. Their framework separates reliance as the probability a human follows an AI recommendation from challenges in forming accurate beliefs about the situation. The framework defines a benchmark for complementary performance representing the maximum attainable performance with AI and human cooperation, and a baseline for performance without cooperation. They apply the framework to three AI-advised decision-making experiments, showing how it can separate loss due to mis-reliance from loss due to inaccurate signal differentiation. The framework evaluates these losses by comparing to a baseline and a benchmark for complementary performance defined by the expected payoff of a rational decision-maker. The framework allows for a continuous assessment of reliance in payoff space, enabling more fine-grained analysis. The authors define appropriate reliance as the rational decision-maker's reliance level on the AI, conditional on the AI recommendation being different from the human recommendation. They also define behavioral under-reliance and over-reliance by comparing the behavioral reliance level to the appropriate reliance level. The framework separates the loss into reliance loss and discrimination loss, where reliance loss is the loss from over- or under-reliance on the AI, and discrimination loss is the loss from not accurately differentiating when the AI is better than the human. The framework is applied to three experiments, demonstrating how it can better reveal the limits of human performance and specific sources of behavioral loss. The authors conclude that the framework provides a valuable tool for evaluating AI reliance and understanding the sources of performance loss in human-AI teams.