This paper investigates the generalization ability of fine-tuned large language models (LLMs) across different tasks and domains. The study explores how fine-tuning affects the models' ability to generalize to new tasks and data. The research focuses on whether fine-tuning on specific tasks, such as generation and classification, leads to different generalization behaviors. The authors conducted extensive experiments across five language tasks using various datasets to analyze the impact of fine-tuning on LLMs' generalization. The study reveals that models fine-tuned on generation tasks often show negative transfer when applied to out-of-domain datasets, while models fine-tuned on classification tasks tend to exhibit positive transfer. Interestingly, integrating in-context learning (ICL) during fine-tuning on generation tasks can enhance the model's generalization ability. The research also finds that fine-tuning on classification tasks may hinder performance on generation tasks, suggesting that the task-specific nature of fine-tuning can limit the model's adaptability. The paper highlights the trade-off between specialization and generalization in fine-tuning. While fine-tuning can improve performance on specific tasks, it may reduce the model's ability to generalize to new tasks and domains. The study also shows that the effectiveness of fine-tuning depends on the task type and the size of the training data. For example, increasing the training data size does not always lead to better performance, indicating that the relationship between training data and generalization is complex and task-dependent. The authors propose that fine-tuning with in-context learning (FTICL) can improve the generalization ability of LLMs for generation tasks by leveraging both fine-tuning and ICL. FTICL models show better performance on out-of-domain tasks compared to vanilla fine-tuned models. However, for classification tasks, FTICL does not consistently improve generalization, suggesting that the effectiveness of FTICL depends on the task type. The study concludes that fine-tuning can significantly affect the generalization ability of LLMs, and the choice of fine-tuning strategy should be tailored to the specific task and domain. The findings provide insights into the evolving landscape of fine-tuning practices for LLMs, emphasizing the importance of balancing specialization and generalization in model training.This paper investigates the generalization ability of fine-tuned large language models (LLMs) across different tasks and domains. The study explores how fine-tuning affects the models' ability to generalize to new tasks and data. The research focuses on whether fine-tuning on specific tasks, such as generation and classification, leads to different generalization behaviors. The authors conducted extensive experiments across five language tasks using various datasets to analyze the impact of fine-tuning on LLMs' generalization. The study reveals that models fine-tuned on generation tasks often show negative transfer when applied to out-of-domain datasets, while models fine-tuned on classification tasks tend to exhibit positive transfer. Interestingly, integrating in-context learning (ICL) during fine-tuning on generation tasks can enhance the model's generalization ability. The research also finds that fine-tuning on classification tasks may hinder performance on generation tasks, suggesting that the task-specific nature of fine-tuning can limit the model's adaptability. The paper highlights the trade-off between specialization and generalization in fine-tuning. While fine-tuning can improve performance on specific tasks, it may reduce the model's ability to generalize to new tasks and domains. The study also shows that the effectiveness of fine-tuning depends on the task type and the size of the training data. For example, increasing the training data size does not always lead to better performance, indicating that the relationship between training data and generalization is complex and task-dependent. The authors propose that fine-tuning with in-context learning (FTICL) can improve the generalization ability of LLMs for generation tasks by leveraging both fine-tuning and ICL. FTICL models show better performance on out-of-domain tasks compared to vanilla fine-tuned models. However, for classification tasks, FTICL does not consistently improve generalization, suggesting that the effectiveness of FTICL depends on the task type. The study concludes that fine-tuning can significantly affect the generalization ability of LLMs, and the choice of fine-tuning strategy should be tailored to the specific task and domain. The findings provide insights into the evolving landscape of fine-tuning practices for LLMs, emphasizing the importance of balancing specialization and generalization in model training.