The paper introduces InsCL, a novel paradigm for fine-tuning large language models (LLMs) with instructions, focusing on addressing the issue of catastrophic forgetting in continual learning. InsCL dynamically replays previous data based on task similarity, calculated using Wasserstein Distance with instructions. It also introduces the Instruction Information Metric (InsInfo) to quantify the complexity and diversity of instructions, guiding the replay process to prioritize high-quality data. Extensive experiments on 16 tasks with different training orders show that InsCL achieves significant performance improvements, with a 3.0 Relative Gain compared to Random Replay and a 27.96 Relative Gain compared to No Replay. The paper also analyzes the forgetting phenomenon, finding that complex reasoning tasks suffer from higher forgetting rates, primarily due to instruction-related issues. The effectiveness of InsCL is demonstrated through its ability to mitigate forgetting and maintain performance stability over multiple tasks.The paper introduces InsCL, a novel paradigm for fine-tuning large language models (LLMs) with instructions, focusing on addressing the issue of catastrophic forgetting in continual learning. InsCL dynamically replays previous data based on task similarity, calculated using Wasserstein Distance with instructions. It also introduces the Instruction Information Metric (InsInfo) to quantify the complexity and diversity of instructions, guiding the replay process to prioritize high-quality data. Extensive experiments on 16 tasks with different training orders show that InsCL achieves significant performance improvements, with a 3.0 Relative Gain compared to Random Replay and a 27.96 Relative Gain compared to No Replay. The paper also analyzes the forgetting phenomenon, finding that complex reasoning tasks suffer from higher forgetting rates, primarily due to instruction-related issues. The effectiveness of InsCL is demonstrated through its ability to mitigate forgetting and maintain performance stability over multiple tasks.