The paper "Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction" addresses the issue of alignment tax, a phenomenon where the performance of large language models (LLMs) on standard knowledge and reasoning benchmarks degrades during supervised fine-tuning (SFT) on instruction-following data. The authors hypothesize that data biases are a major cause of this degradation. To mitigate this, they propose a simple yet effective framework called $\mathcal{O} \mathcal{T} \mathcal{M}$ (Disperse-Then-Merge). This framework involves distributing instruction-following data into multiple portions, training sub-models on each portion, and then merging these sub-models to eliminate data biases. Extensive experiments across various benchmarks show that $\mathcal{O} \mathcal{T} \mathcal{M}$ outperforms sophisticated methods such as data curation and regularization, demonstrating its effectiveness in reducing alignment tax without incurring additional costs. The paper also discusses the robustness of the approach across different data sizes and domains, and compares it with other methods like model ensemble and parameter-efficient fine-tuning techniques.The paper "Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction" addresses the issue of alignment tax, a phenomenon where the performance of large language models (LLMs) on standard knowledge and reasoning benchmarks degrades during supervised fine-tuning (SFT) on instruction-following data. The authors hypothesize that data biases are a major cause of this degradation. To mitigate this, they propose a simple yet effective framework called $\mathcal{O} \mathcal{T} \mathcal{M}$ (Disperse-Then-Merge). This framework involves distributing instruction-following data into multiple portions, training sub-models on each portion, and then merging these sub-models to eliminate data biases. Extensive experiments across various benchmarks show that $\mathcal{O} \mathcal{T} \mathcal{M}$ outperforms sophisticated methods such as data curation and regularization, demonstrating its effectiveness in reducing alignment tax without incurring additional costs. The paper also discusses the robustness of the approach across different data sizes and domains, and compares it with other methods like model ensemble and parameter-efficient fine-tuning techniques.