This paper studies a family of gradient descent algorithms for approximating the regression function from reproducing kernel Hilbert spaces (RKHSs), characterized by a polynomial decreasing step size. By solving a bias-variance trade-off, the paper derives an early stopping rule and probabilistic upper bounds for algorithm convergence. The results are applied to classification, where fast convergence rates for plug-in classifiers are achieved. The paper also connects the methods to boosting, Landweber iterations, and online learning as stochastic approximations of gradient descent. The paper investigates two iteration paths in RKHSs: population iteration (expected risk minimization) and sample iteration (empirical risk minimization). The population iteration converges to the regression function, while the sample iteration often overfits. Keeping these paths close acts as regularization, balancing bias and variance. Early stopping reduces variance but increases bias if too early, and vice versa if too late. Solving this trade-off leads to an optimal early stopping rule. The paper shows that under early stopping, the proposed algorithms converge polynomially to the regression function under regularity assumptions. The constant step size algorithm is the fastest in the family. The convergence rates are suboptimal and can be improved. The results are applied to classification, where fast convergence to the Bayes classifier is achieved under certain noise assumptions. Early stopping regularization is more effective than regularized least squares (e.g., ridge regression) as it avoids the saturation phenomenon, where the rate of convergence plateaus. The algorithms are viewed as randomized discretizations of Landweber iterations in inverse problems. The paper is organized into sections discussing main results, related works, proofs, and applications to classification. It concludes with summaries of findings and open problems. The paper contributes to understanding the convergence of gradient descent algorithms in RKHSs and their applications in learning and inverse problems.This paper studies a family of gradient descent algorithms for approximating the regression function from reproducing kernel Hilbert spaces (RKHSs), characterized by a polynomial decreasing step size. By solving a bias-variance trade-off, the paper derives an early stopping rule and probabilistic upper bounds for algorithm convergence. The results are applied to classification, where fast convergence rates for plug-in classifiers are achieved. The paper also connects the methods to boosting, Landweber iterations, and online learning as stochastic approximations of gradient descent. The paper investigates two iteration paths in RKHSs: population iteration (expected risk minimization) and sample iteration (empirical risk minimization). The population iteration converges to the regression function, while the sample iteration often overfits. Keeping these paths close acts as regularization, balancing bias and variance. Early stopping reduces variance but increases bias if too early, and vice versa if too late. Solving this trade-off leads to an optimal early stopping rule. The paper shows that under early stopping, the proposed algorithms converge polynomially to the regression function under regularity assumptions. The constant step size algorithm is the fastest in the family. The convergence rates are suboptimal and can be improved. The results are applied to classification, where fast convergence to the Bayes classifier is achieved under certain noise assumptions. Early stopping regularization is more effective than regularized least squares (e.g., ridge regression) as it avoids the saturation phenomenon, where the rate of convergence plateaus. The algorithms are viewed as randomized discretizations of Landweber iterations in inverse problems. The paper is organized into sections discussing main results, related works, proofs, and applications to classification. It concludes with summaries of findings and open problems. The paper contributes to understanding the convergence of gradient descent algorithms in RKHSs and their applications in learning and inverse problems.