This paper investigates the phenomenon of mode connectivity in the continual learning (CL) of large language models (LLMs) and proposes a novel method called Interpolation-based LoRA (I-LoRA) to address catastrophic forgetting. The study shows that different minima in the loss landscape of LLMs can be connected by low-loss paths, enabling a balance between learning plasticity and memory stability. I-LoRA constructs a dual-memory framework, consisting of a fast learner for rapid adaptation and a slow learner for preserving long-term knowledge. By interpolating between these two learners, I-LoRA achieves a more optimal trade-off between stability and plasticity. Extensive experiments on eight domain-specific CL benchmarks demonstrate that I-LoRA significantly outperforms existing state-of-the-art methods, achieving up to 11% performance gains. The method is implemented using LoRA parameter interpolations and is shown to effectively mitigate catastrophic forgetting in parameter-efficient fine-tuning scenarios. The results validate the effectiveness of mode connectivity in enhancing CL for LLMs and provide a strong baseline for future research in this area.This paper investigates the phenomenon of mode connectivity in the continual learning (CL) of large language models (LLMs) and proposes a novel method called Interpolation-based LoRA (I-LoRA) to address catastrophic forgetting. The study shows that different minima in the loss landscape of LLMs can be connected by low-loss paths, enabling a balance between learning plasticity and memory stability. I-LoRA constructs a dual-memory framework, consisting of a fast learner for rapid adaptation and a slow learner for preserving long-term knowledge. By interpolating between these two learners, I-LoRA achieves a more optimal trade-off between stability and plasticity. Extensive experiments on eight domain-specific CL benchmarks demonstrate that I-LoRA significantly outperforms existing state-of-the-art methods, achieving up to 11% performance gains. The method is implemented using LoRA parameter interpolations and is shown to effectively mitigate catastrophic forgetting in parameter-efficient fine-tuning scenarios. The results validate the effectiveness of mode connectivity in enhancing CL for LLMs and provide a strong baseline for future research in this area.