The paper introduces MoRAL (Mixture-of-Experts Augmented Low Rank Adaptation for Lifelong learning), a novel approach to enhance the lifelong learning capabilities of large language models (LLMs). MoRAL combines the multi-tasking abilities of Mixture-of-Experts (MoE) with the fine-tuning capabilities of Low Rank Adaptation (LoRA). Unlike traditional methods that use factual triplets as inputs, MoRAL relies on question-answer pairs, which are more practical and effective for robust and efficient learning. The authors introduce the 5L-bench, a new evaluation benchmark that includes a curated dataset of question-answer pairs and evaluation metrics for open-book and closed-book settings. Experimental results show that MoRAL significantly improves LLM performance in open-book settings, outperforms baseline models, and demonstrates better knowledge retention compared to baselines. The study also highlights the importance of larger models in handling context and information filtering, and discusses the limitations of the approach, such as the need for more robust evaluators and the potential for superficial learning.The paper introduces MoRAL (Mixture-of-Experts Augmented Low Rank Adaptation for Lifelong learning), a novel approach to enhance the lifelong learning capabilities of large language models (LLMs). MoRAL combines the multi-tasking abilities of Mixture-of-Experts (MoE) with the fine-tuning capabilities of Low Rank Adaptation (LoRA). Unlike traditional methods that use factual triplets as inputs, MoRAL relies on question-answer pairs, which are more practical and effective for robust and efficient learning. The authors introduce the 5L-bench, a new evaluation benchmark that includes a curated dataset of question-answer pairs and evaluation metrics for open-book and closed-book settings. Experimental results show that MoRAL significantly improves LLM performance in open-book settings, outperforms baseline models, and demonstrates better knowledge retention compared to baselines. The study also highlights the importance of larger models in handling context and information filtering, and discusses the limitations of the approach, such as the need for more robust evaluators and the potential for superficial learning.