This paper provides the first theoretical analysis of the Mixture-of-Experts (MoE) model in the context of continual learning (CL). The authors address the issue of catastrophic forgetting, where models forget old tasks as they learn new ones. MoE is shown to mitigate this problem by distributing tasks among multiple experts, each specialized in different tasks. The key contributions of the paper include: 1. **Theoretical Analysis**: The authors establish that MoE can effectively diversify experts and balance their loads, leading to better performance in CL. They prove that after sufficient training rounds, each expert specializes in a specific task, and the router consistently selects the right expert for each task. 2. **Early Termination**: An important finding is that the gating network should be terminated after a certain number of training rounds to ensure system convergence. This is necessary because continuous updates to the gating network can lead to incorrect task routing and increased forgetting. 3. **Forgetting and Generalization**: The paper provides explicit expressions for the expected forgetting and overall generalization error, showing that MoE significantly reduces these metrics compared to a single expert. Adding more experts does not necessarily enhance performance but delays convergence. 4. **Experiments**: Extensive experiments on synthetic and real datasets (MNIST) validate the theoretical findings. The results show that MoE improves CL performance, and the insights from linear models extend to deep neural networks (DNNs). The paper concludes by highlighting the practical implications of these findings for designing effective MoE models in CL.This paper provides the first theoretical analysis of the Mixture-of-Experts (MoE) model in the context of continual learning (CL). The authors address the issue of catastrophic forgetting, where models forget old tasks as they learn new ones. MoE is shown to mitigate this problem by distributing tasks among multiple experts, each specialized in different tasks. The key contributions of the paper include: 1. **Theoretical Analysis**: The authors establish that MoE can effectively diversify experts and balance their loads, leading to better performance in CL. They prove that after sufficient training rounds, each expert specializes in a specific task, and the router consistently selects the right expert for each task. 2. **Early Termination**: An important finding is that the gating network should be terminated after a certain number of training rounds to ensure system convergence. This is necessary because continuous updates to the gating network can lead to incorrect task routing and increased forgetting. 3. **Forgetting and Generalization**: The paper provides explicit expressions for the expected forgetting and overall generalization error, showing that MoE significantly reduces these metrics compared to a single expert. Adding more experts does not necessarily enhance performance but delays convergence. 4. **Experiments**: Extensive experiments on synthetic and real datasets (MNIST) validate the theoretical findings. The results show that MoE improves CL performance, and the insights from linear models extend to deep neural networks (DNNs). The paper concludes by highlighting the practical implications of these findings for designing effective MoE models in CL.