This paper presents a statistical analysis of regularization-based continual learning in linear regression tasks. The authors focus on how different regularization terms affect model performance and derive the convergence rate for the oracle estimator, which assumes all data is available simultaneously. They consider a family of generalized $\ell_2$-regularization algorithms indexed by matrix-valued hyperparameters, which include the minimum norm estimator and continual ridge regression as special cases. As more tasks are introduced, they derive an iterative update formula for the estimation error of generalized $\ell_2$-regularized estimators, allowing them to determine the optimal hyperparameters. The choice of hyperparameters can balance forward and backward knowledge transfer and adjust for data heterogeneity. The estimation error of the optimal algorithm is shown to be of the same order as that of the oracle estimator, while the minimum norm estimator and continual ridge regression are found to be suboptimal. The paper also establishes the equivalence between early stopping and generalized $\ell_2$-regularization in continual learning. Experiments are conducted to complement the theoretical findings, showing that the generalized $\ell_2$-regularized estimator achieves performance comparable to the oracle estimator. The results demonstrate that the optimal hyperparameters allow for a balanced trade-off between forward and backward knowledge transfer, and that the estimation error of the generalized $\ell_2$-regularized estimator matches the order of the oracle estimator. The paper also shows that early stopping and generalized $\ell_2$-regularization are equivalent in the context of continual learning.This paper presents a statistical analysis of regularization-based continual learning in linear regression tasks. The authors focus on how different regularization terms affect model performance and derive the convergence rate for the oracle estimator, which assumes all data is available simultaneously. They consider a family of generalized $\ell_2$-regularization algorithms indexed by matrix-valued hyperparameters, which include the minimum norm estimator and continual ridge regression as special cases. As more tasks are introduced, they derive an iterative update formula for the estimation error of generalized $\ell_2$-regularized estimators, allowing them to determine the optimal hyperparameters. The choice of hyperparameters can balance forward and backward knowledge transfer and adjust for data heterogeneity. The estimation error of the optimal algorithm is shown to be of the same order as that of the oracle estimator, while the minimum norm estimator and continual ridge regression are found to be suboptimal. The paper also establishes the equivalence between early stopping and generalized $\ell_2$-regularization in continual learning. Experiments are conducted to complement the theoretical findings, showing that the generalized $\ell_2$-regularized estimator achieves performance comparable to the oracle estimator. The results demonstrate that the optimal hyperparameters allow for a balanced trade-off between forward and backward knowledge transfer, and that the estimation error of the generalized $\ell_2$-regularized estimator matches the order of the oracle estimator. The paper also shows that early stopping and generalized $\ell_2$-regularization are equivalent in the context of continual learning.