22 Oct 2020 | Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, Simone Calderara
The paper introduces a novel baseline for Continual Learning (CL) called Dark Experience Replay (DER), which aims to address the challenges of General Continual Learning (GCL) where task boundaries are blurred and domain and class distributions shift gradually or suddenly. DER combines rehearsal with knowledge distillation and regularization, sampling the network's logits throughout the optimization trajectory to promote consistency with past experiences. The authors demonstrate that DER outperforms existing consolidated approaches in standard benchmarks and a novel GCL evaluation setting (MNIST-560), showing its effectiveness in handling limited resources and complex scenarios. They also explore the generalization capabilities of DER, finding that it exhibits beneficial regularization properties beyond mere performance improvement. The code for DER is available at https://github.com/aimagelab/mammoth.The paper introduces a novel baseline for Continual Learning (CL) called Dark Experience Replay (DER), which aims to address the challenges of General Continual Learning (GCL) where task boundaries are blurred and domain and class distributions shift gradually or suddenly. DER combines rehearsal with knowledge distillation and regularization, sampling the network's logits throughout the optimization trajectory to promote consistency with past experiences. The authors demonstrate that DER outperforms existing consolidated approaches in standard benchmarks and a novel GCL evaluation setting (MNIST-560), showing its effectiveness in handling limited resources and complex scenarios. They also explore the generalization capabilities of DER, finding that it exhibits beneficial regularization properties beyond mere performance improvement. The code for DER is available at https://github.com/aimagelab/mammoth.