The paper "The Reasons that Agents Act: Intention and Instrumental Goals" by Francis Rhys Ward, Matt MacDermott, Francesca Toni, Francesco Belardinelli, and Tom Everitt explores the concept of intention in AI agents, which is crucial for understanding and designing safe AI systems. The authors operationalize the notion of intention in structural causal influence models (SCIMs), a framework that integrates causality and decision-making. They introduce a formal definition of intention that captures the reasons an agent chooses its decisions, including instrumental goals, which are key in safe AI agent design. The paper outlines three stages of operationalizing intention: 1. **Intention to Cause an Outcome**: An agent intends to cause an outcome if guaranteeing another action would make that action equally good for the agent. 2. **Intention to Cause Multiple Outcomes**: An agent can intend multiple outcomes if they are part of a superset that is intended. 3. **Intention in a Random Setting**: An agent intends to cause outcomes in specific settings if the outcomes are caused by their actions in those settings. The authors formalize these operationalizations using SCIMs, which model causality and decision-making. They demonstrate that their definition captures the intuitive notion of intent and satisfies key criteria for algorithmic intent. The paper also relates their definition to past concepts, such as actual causality and instrumental control incentives (ICIs), showing that their notion of intention shares graphical criteria with ICIs. Additionally, the authors propose a behavioral notion of intent, which is equivalent to the subjective notion if the agent is robustly optimal. This allows for inferring the intentions of reinforcement learning agents and language models from their behavior without needing to know their utility functions. The paper concludes with a discussion on assessing the intentions of real-world ML systems, including a case study on GPT-4, highlighting the challenges and limitations of their behavioral definition.The paper "The Reasons that Agents Act: Intention and Instrumental Goals" by Francis Rhys Ward, Matt MacDermott, Francesca Toni, Francesco Belardinelli, and Tom Everitt explores the concept of intention in AI agents, which is crucial for understanding and designing safe AI systems. The authors operationalize the notion of intention in structural causal influence models (SCIMs), a framework that integrates causality and decision-making. They introduce a formal definition of intention that captures the reasons an agent chooses its decisions, including instrumental goals, which are key in safe AI agent design. The paper outlines three stages of operationalizing intention: 1. **Intention to Cause an Outcome**: An agent intends to cause an outcome if guaranteeing another action would make that action equally good for the agent. 2. **Intention to Cause Multiple Outcomes**: An agent can intend multiple outcomes if they are part of a superset that is intended. 3. **Intention in a Random Setting**: An agent intends to cause outcomes in specific settings if the outcomes are caused by their actions in those settings. The authors formalize these operationalizations using SCIMs, which model causality and decision-making. They demonstrate that their definition captures the intuitive notion of intent and satisfies key criteria for algorithmic intent. The paper also relates their definition to past concepts, such as actual causality and instrumental control incentives (ICIs), showing that their notion of intention shares graphical criteria with ICIs. Additionally, the authors propose a behavioral notion of intent, which is equivalent to the subjective notion if the agent is robustly optimal. This allows for inferring the intentions of reinforcement learning agents and language models from their behavior without needing to know their utility functions. The paper concludes with a discussion on assessing the intentions of real-world ML systems, including a case study on GPT-4, highlighting the challenges and limitations of their behavioral definition.