This paper addresses the issue of catastrophic forgetting in large language models (LLMs) during continuous fine-tuning on diverse domain-specific downstream tasks. The authors investigate the geometric connections between different minima in the loss landscape, revealing that these minima can be connected by a low-loss valley, a phenomenon known as mode connectivity. They propose Interpolation-based LoRA (I-LoRA), a method that leverages this mode connectivity to balance plasticity and stability. I-LoRA constructs a dual-memory framework with a fast learner for quick adaptation and a slow learner for long-term memory preservation. Extensive experiments on eight domain-specific benchmarks demonstrate that I-LoRA significantly improves performance over state-of-the-art methods, achieving up to 11% accuracy gains. The paper provides a strong baseline and insights for future research on LLMs' continual learning.This paper addresses the issue of catastrophic forgetting in large language models (LLMs) during continuous fine-tuning on diverse domain-specific downstream tasks. The authors investigate the geometric connections between different minima in the loss landscape, revealing that these minima can be connected by a low-loss valley, a phenomenon known as mode connectivity. They propose Interpolation-based LoRA (I-LoRA), a method that leverages this mode connectivity to balance plasticity and stability. I-LoRA constructs a dual-memory framework with a fast learner for quick adaptation and a slow learner for long-term memory preservation. Extensive experiments on eight domain-specific benchmarks demonstrate that I-LoRA significantly improves performance over state-of-the-art methods, achieving up to 11% accuracy gains. The paper provides a strong baseline and insights for future research on LLMs' continual learning.