The paper "Orca-Math: Unlocking the potential of SLMs in Grade School Math" by Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah from Microsoft Research presents a 7-billion-parameter SLM (Small Language Model) called Orca-Math, which achieves 86.81% accuracy on the GSM8K benchmark for solving mathematical word problems. This performance is achieved without the need for multiple model calls, external tools, or ensembling, which are common practices in other models like LLAMA-2-70B and WizardMath-70B. The key elements of Orca-Math include: 1. **High-Quality Synthetic Dataset**: A dataset of 200K math problems created using a multi-agent setup where agents collaborate to generate diverse and challenging problems. 2. **Iterative Learning**: The model is trained through supervised fine-tuning and then allowed to practice generating solutions, receiving feedback from a teacher model (GPT4-Turbo) to improve its performance. The paper also discusses the challenges and improvements in training SLMs for mathematical reasoning, highlighting the benefits of high-quality synthetic data and iterative learning. The results show that Orca-Math outperforms significantly larger models while using much smaller datasets, demonstrating the potential of SLMs in solving complex mathematical tasks.The paper "Orca-Math: Unlocking the potential of SLMs in Grade School Math" by Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah from Microsoft Research presents a 7-billion-parameter SLM (Small Language Model) called Orca-Math, which achieves 86.81% accuracy on the GSM8K benchmark for solving mathematical word problems. This performance is achieved without the need for multiple model calls, external tools, or ensembling, which are common practices in other models like LLAMA-2-70B and WizardMath-70B. The key elements of Orca-Math include: 1. **High-Quality Synthetic Dataset**: A dataset of 200K math problems created using a multi-agent setup where agents collaborate to generate diverse and challenging problems. 2. **Iterative Learning**: The model is trained through supervised fine-tuning and then allowed to practice generating solutions, receiving feedback from a teacher model (GPT4-Turbo) to improve its performance. The paper also discusses the challenges and improvements in training SLMs for mathematical reasoning, highlighting the benefits of high-quality synthetic data and iterative learning. The results show that Orca-Math outperforms significantly larger models while using much smaller datasets, demonstrating the potential of SLMs in solving complex mathematical tasks.