This paper introduces a novel framework called Disperse-Then-Merge (DTM) to address the alignment tax issue in instruction tuning for large language models (LLMs). The alignment tax refers to the phenomenon where the performance of LLMs on standard knowledge and reasoning benchmarks deteriorates as the size of instruction-following data increases. The authors hypothesize that this issue is caused by data biases in the instruction-following data, which are learned by the models during training. To mitigate this, the DTM framework disperses the instruction-following data into multiple clusters, trains separate sub-models on each cluster, and then merges these sub-models to reduce the impact of data biases. The DTM framework is simple and effective, outperforming various sophisticated methods such as data curation and training regularization on multiple benchmarks. The framework is evaluated on nine benchmarks involving math reasoning, world knowledge, and code generation, and it shows superior performance compared to existing methods. The results indicate that DTM not only maintains the basic knowledge and reasoning ability of the language model but also improves the instruction-following ability. The study also explores the effectiveness of DTM across different instruction-following data sizes and domains, demonstrating its robustness and generalization. The framework is shown to work with various base LLMs, including Mistral-7b and Baichuan-2-7b. The results suggest that DTM is agnostic to the base LLM and can be applied to more capable models. The paper concludes that the alignment tax is primarily caused by data biases in the instruction-following data, and the DTM framework effectively mitigates this issue by dispersing and merging sub-models. The study highlights the importance of understanding and addressing the root causes of alignment tax to improve the performance of LLMs in real-world applications.This paper introduces a novel framework called Disperse-Then-Merge (DTM) to address the alignment tax issue in instruction tuning for large language models (LLMs). The alignment tax refers to the phenomenon where the performance of LLMs on standard knowledge and reasoning benchmarks deteriorates as the size of instruction-following data increases. The authors hypothesize that this issue is caused by data biases in the instruction-following data, which are learned by the models during training. To mitigate this, the DTM framework disperses the instruction-following data into multiple clusters, trains separate sub-models on each cluster, and then merges these sub-models to reduce the impact of data biases. The DTM framework is simple and effective, outperforming various sophisticated methods such as data curation and training regularization on multiple benchmarks. The framework is evaluated on nine benchmarks involving math reasoning, world knowledge, and code generation, and it shows superior performance compared to existing methods. The results indicate that DTM not only maintains the basic knowledge and reasoning ability of the language model but also improves the instruction-following ability. The study also explores the effectiveness of DTM across different instruction-following data sizes and domains, demonstrating its robustness and generalization. The framework is shown to work with various base LLMs, including Mistral-7b and Baichuan-2-7b. The results suggest that DTM is agnostic to the base LLM and can be applied to more capable models. The paper concludes that the alignment tax is primarily caused by data biases in the instruction-following data, and the DTM framework effectively mitigates this issue by dispersing and merging sub-models. The study highlights the importance of understanding and addressing the root causes of alignment tax to improve the performance of LLMs in real-world applications.