The paper by Gido M. van de Ven and Andreas S. Tolias discusses the challenges of continual or lifelong learning in artificial neural networks, which often suffer from catastrophic forgetting. To address this issue, the authors propose three distinct scenarios for continual learning based on whether task identity is provided at test time and whether the model must infer task identity when it is not provided. These scenarios are: 1. **Task-Incremental Learning (Task-IL)**: Task identity is always provided, and models are trained with task-specific components. 2. **Domain-Incremental Learning (Domain-IL)**: Task identity is not provided at test time, but models do not need to infer task identity. 3. **Class-Incremental Learning (Class-IL)**: Models must solve each task and infer task identity. The authors use the split and permuted MNIST task protocols to compare the performance of various continual learning methods, including regularization-based approaches (e.g., Elastic Weight Consolidation) and replay-based approaches. They find that: - Regularization-based methods struggle in the Class-IL scenario and fail completely. - Replay-based methods (e.g., Learning without Forgetting, Deep Generative Replay, and iCaRL) perform well in all three scenarios. - Task identity is crucial in the Domain-IL scenario, while it is less important in the Task-IL scenario. - The difficulty of the scenarios varies, with the Class-IL scenario being the most challenging. The study highlights the importance of task identity and the need for replay in addressing catastrophic forgetting, especially in more complex scenarios. The authors also provide detailed experimental details and code for reproducibility.The paper by Gido M. van de Ven and Andreas S. Tolias discusses the challenges of continual or lifelong learning in artificial neural networks, which often suffer from catastrophic forgetting. To address this issue, the authors propose three distinct scenarios for continual learning based on whether task identity is provided at test time and whether the model must infer task identity when it is not provided. These scenarios are: 1. **Task-Incremental Learning (Task-IL)**: Task identity is always provided, and models are trained with task-specific components. 2. **Domain-Incremental Learning (Domain-IL)**: Task identity is not provided at test time, but models do not need to infer task identity. 3. **Class-Incremental Learning (Class-IL)**: Models must solve each task and infer task identity. The authors use the split and permuted MNIST task protocols to compare the performance of various continual learning methods, including regularization-based approaches (e.g., Elastic Weight Consolidation) and replay-based approaches. They find that: - Regularization-based methods struggle in the Class-IL scenario and fail completely. - Replay-based methods (e.g., Learning without Forgetting, Deep Generative Replay, and iCaRL) perform well in all three scenarios. - Task identity is crucial in the Domain-IL scenario, while it is less important in the Task-IL scenario. - The difficulty of the scenarios varies, with the Class-IL scenario being the most challenging. The study highlights the importance of task identity and the need for replay in addressing catastrophic forgetting, especially in more complex scenarios. The authors also provide detailed experimental details and code for reproducibility.