**Magic: Generating Self-Correction Guideline for In-Context Text-to-SQL** **Authors:** Arian Askari, Christian Poelitz, Xinye Tang **Institution:** Leiden University, Microsoft **Abstract:** Self-correction in text-to-SQL involves prompting large language models (LLMs) to revise incorrectly generated SQL queries. Traditional methods rely on manually crafted guidelines, which are labor-intensive and limited by human expertise. This paper introduces MAGIC, a novel multi-agent method that automates the creation of self-correction guidelines. MAGIC employs three specialized agents—manager, correction, and feedback agents—to collaboratively generate and refine guidelines tailored to LLM mistakes. Extensive experiments show that MAGIC's guidelines outperform those created by human experts, enhancing the interpretability of corrections and providing insights into LLM failures and successes. **Introduction:** Text-to-SQL is crucial for non-expert data analysts to extract information from relational databases using natural language. While LLMs have improved significantly, they still make mistakes, with GPT4 having a notable accuracy gap of 30% compared to humans. Self-correction aims to address these errors by prompting LLMs to revise their incorrect SQL queries. Existing methods often rely on human-engineered guidelines, which are time-consuming and limited in scope. MAGIC addresses this by automatically generating effective self-correction guidelines, improving the effectiveness of LLM-based text-to-SQL methods. **MAGIC:** MAGIC consists of three agents: 1. **Manager:** Manages the feedback and correction cycles, integrating feedback from the feedback agent to revise incorrect SQL queries. 2. **Feedback Agent:** Provides explanations for mistakes in predicted SQL queries. 3. **Correction Agent:** Generates revised SQL queries based on feedback. **Key Contributions:** - Introduces MAGIC, a novel multi-agent method for generating self-correction guidelines. - Outperforms human-written guidelines in improving the effectiveness of LLM-based text-to-SQL methods. - Enhances interpretability of corrections, providing insights into LLM failures and successes. **Experimental Setup:** - **Datasets:** Spider and BIRD datasets. - **Metrics:** Execution Accuracy (EX) and Valid Efficiency Score (VES). - **Baselines:** Reproduced DIN-SQL and compared with various self-correction methods. **Results:** - MAGIC's guidelines significantly improve execution accuracy and efficiency compared to existing baselines. - The effectiveness of MAGIC's guidelines is influenced by the quantity of feedback, with 10 batches of feedback optimal. - The manager agent significantly reduces the number of iterations and increases the number of corrected SQL queries. **Discussion:** - MAGIC's guidelines are applicable to other LLM-based methods and can be adapted to different database difficulties. - The guidelines do not directly copy examples from feedback but aggregate multiple pieces of feedback to create new examples. **Conclusion:** MAGIC offers a novel approach to self-correction in text-to-SQL**Magic: Generating Self-Correction Guideline for In-Context Text-to-SQL** **Authors:** Arian Askari, Christian Poelitz, Xinye Tang **Institution:** Leiden University, Microsoft **Abstract:** Self-correction in text-to-SQL involves prompting large language models (LLMs) to revise incorrectly generated SQL queries. Traditional methods rely on manually crafted guidelines, which are labor-intensive and limited by human expertise. This paper introduces MAGIC, a novel multi-agent method that automates the creation of self-correction guidelines. MAGIC employs three specialized agents—manager, correction, and feedback agents—to collaboratively generate and refine guidelines tailored to LLM mistakes. Extensive experiments show that MAGIC's guidelines outperform those created by human experts, enhancing the interpretability of corrections and providing insights into LLM failures and successes. **Introduction:** Text-to-SQL is crucial for non-expert data analysts to extract information from relational databases using natural language. While LLMs have improved significantly, they still make mistakes, with GPT4 having a notable accuracy gap of 30% compared to humans. Self-correction aims to address these errors by prompting LLMs to revise their incorrect SQL queries. Existing methods often rely on human-engineered guidelines, which are time-consuming and limited in scope. MAGIC addresses this by automatically generating effective self-correction guidelines, improving the effectiveness of LLM-based text-to-SQL methods. **MAGIC:** MAGIC consists of three agents: 1. **Manager:** Manages the feedback and correction cycles, integrating feedback from the feedback agent to revise incorrect SQL queries. 2. **Feedback Agent:** Provides explanations for mistakes in predicted SQL queries. 3. **Correction Agent:** Generates revised SQL queries based on feedback. **Key Contributions:** - Introduces MAGIC, a novel multi-agent method for generating self-correction guidelines. - Outperforms human-written guidelines in improving the effectiveness of LLM-based text-to-SQL methods. - Enhances interpretability of corrections, providing insights into LLM failures and successes. **Experimental Setup:** - **Datasets:** Spider and BIRD datasets. - **Metrics:** Execution Accuracy (EX) and Valid Efficiency Score (VES). - **Baselines:** Reproduced DIN-SQL and compared with various self-correction methods. **Results:** - MAGIC's guidelines significantly improve execution accuracy and efficiency compared to existing baselines. - The effectiveness of MAGIC's guidelines is influenced by the quantity of feedback, with 10 batches of feedback optimal. - The manager agent significantly reduces the number of iterations and increases the number of corrected SQL queries. **Discussion:** - MAGIC's guidelines are applicable to other LLM-based methods and can be adapted to different database difficulties. - The guidelines do not directly copy examples from feedback but aggregate multiple pieces of feedback to create new examples. **Conclusion:** MAGIC offers a novel approach to self-correction in text-to-SQL