The paper introduces Self-Distillation Fine-Tuning (SDFT), a novel approach to address the distribution gap between task datasets and Large Language Models (LLMs) during fine-tuning. The distribution gap is identified as the primary cause of challenges in balancing performance and preserving general instruction-following abilities. SDFT guides fine-tuning by using a distilled dataset generated by the model itself, which matches the original distribution of the LLMs. This method effectively mitigates catastrophic forgetting while achieving comparable or superior performance on downstream tasks compared to vanilla fine-tuning. Experimental results on the Llama-2-chat model across various benchmarks demonstrate the effectiveness of SDFT. The code for SDFT is available at <https://github.com/sail-sg/sdft>.The paper introduces Self-Distillation Fine-Tuning (SDFT), a novel approach to address the distribution gap between task datasets and Large Language Models (LLMs) during fine-tuning. The distribution gap is identified as the primary cause of challenges in balancing performance and preserving general instruction-following abilities. SDFT guides fine-tuning by using a distilled dataset generated by the model itself, which matches the original distribution of the LLMs. This method effectively mitigates catastrophic forgetting while achieving comparable or superior performance on downstream tasks compared to vanilla fine-tuning. Experimental results on the Llama-2-chat model across various benchmarks demonstrate the effectiveness of SDFT. The code for SDFT is available at <https://github.com/sail-sg/sdft>.