This paper introduces a unified and general framework for Continual Learning (CL), addressing the challenge of retaining previously acquired knowledge while learning new data. The framework encompasses various existing CL methods, including regularization-based, Bayesian-based, and memory-replay-based techniques, by formulating them within a common optimization objective. This objective, defined using Bregman divergences, allows for the integration of different approaches into a cohesive framework. The paper also introduces a novel concept called *refresh learning*, inspired by neuroscience, which involves unlearning and then relearning current data to enhance CL performance. *Refresh learning* is designed as a versatile plug-in that can be seamlessly integrated with existing CL methods, improving their adaptability and effectiveness. Extensive experiments on various datasets demonstrate the effectiveness of *refresh learning*, showing significant improvements in overall accuracy and backward transfer compared to baseline methods. The theoretical analysis further supports the effectiveness of *refresh learning* by showing that it approximately minimizes the Fisher Information Matrix (FIM) weighted gradient norm of the loss function, leading to a flatter loss landscape and better generalization.This paper introduces a unified and general framework for Continual Learning (CL), addressing the challenge of retaining previously acquired knowledge while learning new data. The framework encompasses various existing CL methods, including regularization-based, Bayesian-based, and memory-replay-based techniques, by formulating them within a common optimization objective. This objective, defined using Bregman divergences, allows for the integration of different approaches into a cohesive framework. The paper also introduces a novel concept called *refresh learning*, inspired by neuroscience, which involves unlearning and then relearning current data to enhance CL performance. *Refresh learning* is designed as a versatile plug-in that can be seamlessly integrated with existing CL methods, improving their adaptability and effectiveness. Extensive experiments on various datasets demonstrate the effectiveness of *refresh learning*, showing significant improvements in overall accuracy and backward transfer compared to baseline methods. The theoretical analysis further supports the effectiveness of *refresh learning* by showing that it approximately minimizes the Fisher Information Matrix (FIM) weighted gradient norm of the loss function, leading to a flatter loss landscape and better generalization.