The paper introduces MoCL, a novel modular and compositional continual learning framework for language models. MoCL aims to address the challenges of catastrophic forgetting and effective knowledge transfer without relying on rehearsal or storing additional data. The framework continuously adds new task-specific modules to language models and composes them with existing modules based on task matching weights. This approach avoids catastrophic forgetting by freezing task-specific modules after training on a task and facilitates knowledge transfer by reusing previously learned knowledge in a weighted manner. Experiments on various benchmarks, including near-domain and far-domain continual learning tasks, demonstrate that MoCL outperforms state-of-the-art methods, effectively handling knowledge transfer and maintaining performance across different task similarities. The paper also discusses the limitations of the current work, such as the scope of evaluation and the potential for module pruning to improve efficiency in more extensive continual learning scenarios.The paper introduces MoCL, a novel modular and compositional continual learning framework for language models. MoCL aims to address the challenges of catastrophic forgetting and effective knowledge transfer without relying on rehearsal or storing additional data. The framework continuously adds new task-specific modules to language models and composes them with existing modules based on task matching weights. This approach avoids catastrophic forgetting by freezing task-specific modules after training on a task and facilitates knowledge transfer by reusing previously learned knowledge in a weighted manner. Experiments on various benchmarks, including near-domain and far-domain continual learning tasks, demonstrate that MoCL outperforms state-of-the-art methods, effectively handling knowledge transfer and maintaining performance across different task similarities. The paper also discusses the limitations of the current work, such as the scope of evaluation and the potential for module pruning to improve efficiency in more extensive continual learning scenarios.