This paper investigates the impact of fine-tuning on the generalization ability of Large Language Models (LLMs). The authors conduct extensive experiments across five distinct language tasks on various datasets to explore whether fine-tuning affects the LLMs' generalization ability. The main findings reveal that models fine-tuned on generation tasks exhibit different behaviors in generalizing to different domains and tasks compared to those fine-tuned on classification tasks. Specifically, models fine-tuned for classification tasks tend to show positive transfer when applied to out-of-domain datasets of the same task type, while models fine-tuned for generation tasks often experience negative transfer under similar conditions. The study also finds that integrating in-context learning (ICL) during fine-tuning on generation tasks can enhance the model's generalization ability. The authors aim to contribute valuable insights into the evolving landscape of fine-tuning practices for LLMs, particularly in enhancing task-specific performance and fostering broader generalization abilities. The code and data are available at <https://github.com/LHRYANG/Generalization_of_FT-LLM>.This paper investigates the impact of fine-tuning on the generalization ability of Large Language Models (LLMs). The authors conduct extensive experiments across five distinct language tasks on various datasets to explore whether fine-tuning affects the LLMs' generalization ability. The main findings reveal that models fine-tuned on generation tasks exhibit different behaviors in generalizing to different domains and tasks compared to those fine-tuned on classification tasks. Specifically, models fine-tuned for classification tasks tend to show positive transfer when applied to out-of-domain datasets of the same task type, while models fine-tuned for generation tasks often experience negative transfer under similar conditions. The study also finds that integrating in-context learning (ICL) during fine-tuning on generation tasks can enhance the model's generalization ability. The authors aim to contribute valuable insights into the evolving landscape of fine-tuning practices for LLMs, particularly in enhancing task-specific performance and fostering broader generalization abilities. The code and data are available at <https://github.com/LHRYANG/Generalization_of_FT-LLM>.