This paper investigates whether transformers perform case-based or rule-based reasoning when solving math problems. Despite their impressive performance in complex tasks, transformers struggle with simple arithmetic, such as addition, often relying on similar examples from training data rather than learning underlying rules. The authors define two reasoning mechanisms: case-based reasoning, which depends on similar examples, and rule-based reasoning, which involves learning and applying general rules. Through intervention experiments on five math tasks, they confirm that transformers use case-based reasoning, aligning with previous findings that transformers rely on subgraph matching or shortcut learning. To improve this, the authors propose Rule-Following Fine-Tuning (RFFT), a technique that explicitly teaches transformers to follow rules step by step. By providing clear rules in the input and instructing the model to recite and apply them, RFFT enables LLMs to generalize better. For example, LLMs fine-tuned on 1-5 digit addition achieve over 95% accuracy on 12-digit addition, significantly outperforming scratchpad methods. This demonstrates that explicitly teaching rules helps LLMs learn rule-based reasoning and generalize better. The study also shows that case-based reasoning is sensitive to data splits, with performance dropping when training data is removed. In contrast, rule-based reasoning remains stable. The results highlight the importance of rule-based reasoning for systematic generalization. The paper concludes that teaching LLMs to follow explicit rules can enhance their ability to solve complex tasks, similar to how humans learn arithmetic. The findings contribute to understanding how LLMs reason and offer a practical method to improve their generalization capabilities.This paper investigates whether transformers perform case-based or rule-based reasoning when solving math problems. Despite their impressive performance in complex tasks, transformers struggle with simple arithmetic, such as addition, often relying on similar examples from training data rather than learning underlying rules. The authors define two reasoning mechanisms: case-based reasoning, which depends on similar examples, and rule-based reasoning, which involves learning and applying general rules. Through intervention experiments on five math tasks, they confirm that transformers use case-based reasoning, aligning with previous findings that transformers rely on subgraph matching or shortcut learning. To improve this, the authors propose Rule-Following Fine-Tuning (RFFT), a technique that explicitly teaches transformers to follow rules step by step. By providing clear rules in the input and instructing the model to recite and apply them, RFFT enables LLMs to generalize better. For example, LLMs fine-tuned on 1-5 digit addition achieve over 95% accuracy on 12-digit addition, significantly outperforming scratchpad methods. This demonstrates that explicitly teaching rules helps LLMs learn rule-based reasoning and generalize better. The study also shows that case-based reasoning is sensitive to data splits, with performance dropping when training data is removed. In contrast, rule-based reasoning remains stable. The results highlight the importance of rule-based reasoning for systematic generalization. The paper concludes that teaching LLMs to follow explicit rules can enhance their ability to solve complex tasks, similar to how humans learn arithmetic. The findings contribute to understanding how LLMs reason and offer a practical method to improve their generalization capabilities.