This paper presents the Stochastic Average Gradient (SAG) method for minimizing finite sums of smooth convex functions. The SAG method improves upon stochastic gradient (SG) methods by incorporating memory of previous gradient values, achieving faster convergence rates. For convex problems, SAG achieves a convergence rate of $ O(1/k) $, which is faster than the $ O(1/\sqrt{k}) $ rate of SG methods. For strongly-convex problems, SAG achieves a linear convergence rate of $ O(\rho^k) $, which is faster than the sublinear rate of SG methods. Additionally, SAG often outperforms deterministic gradient methods in terms of the number of gradient evaluations. The SAG method has an iteration cost independent of the number of terms in the sum, similar to SG methods, but achieves the convergence rates of deterministic gradient methods. The SAG iterations take the form $ x^{k+1} = x^k - \frac{\alpha_k}{n} \sum_{i=1}^n y_i^k $, where $ y_i^k $ is the gradient of the function at the most recent iteration for index $ i $, or the previous value if the index was not sampled in the current iteration. This allows SAG to maintain a low iteration cost while achieving faster convergence. The paper also discusses related methods, including momentum, gradient averaging, iterate averaging, and stochastic versions of full gradient methods. It shows that SAG outperforms these methods in terms of convergence rates and practical performance. The paper also presents a convergence analysis of the SAG method under standard assumptions, showing that it achieves $ O(1/k) $ convergence for convex problems and linear convergence for strongly-convex problems. The analysis also shows that SAG can be more robust to step size selection and can achieve faster convergence rates with appropriate initialization. The paper also discusses practical implementation details of the SAG method, including structured gradients, just-in-time parameter updates, regularization, warm starting, larger step sizes, line-search when L is not known, mini-batches for vectorized computation, and non-uniform example selection. These details show that SAG can be efficiently implemented for large-scale problems and can be adapted to various problem structures. The paper concludes that SAG is a powerful method for minimizing finite sums of smooth convex functions, with faster convergence rates and better practical performance compared to SG and deterministic gradient methods.This paper presents the Stochastic Average Gradient (SAG) method for minimizing finite sums of smooth convex functions. The SAG method improves upon stochastic gradient (SG) methods by incorporating memory of previous gradient values, achieving faster convergence rates. For convex problems, SAG achieves a convergence rate of $ O(1/k) $, which is faster than the $ O(1/\sqrt{k}) $ rate of SG methods. For strongly-convex problems, SAG achieves a linear convergence rate of $ O(\rho^k) $, which is faster than the sublinear rate of SG methods. Additionally, SAG often outperforms deterministic gradient methods in terms of the number of gradient evaluations. The SAG method has an iteration cost independent of the number of terms in the sum, similar to SG methods, but achieves the convergence rates of deterministic gradient methods. The SAG iterations take the form $ x^{k+1} = x^k - \frac{\alpha_k}{n} \sum_{i=1}^n y_i^k $, where $ y_i^k $ is the gradient of the function at the most recent iteration for index $ i $, or the previous value if the index was not sampled in the current iteration. This allows SAG to maintain a low iteration cost while achieving faster convergence. The paper also discusses related methods, including momentum, gradient averaging, iterate averaging, and stochastic versions of full gradient methods. It shows that SAG outperforms these methods in terms of convergence rates and practical performance. The paper also presents a convergence analysis of the SAG method under standard assumptions, showing that it achieves $ O(1/k) $ convergence for convex problems and linear convergence for strongly-convex problems. The analysis also shows that SAG can be more robust to step size selection and can achieve faster convergence rates with appropriate initialization. The paper also discusses practical implementation details of the SAG method, including structured gradients, just-in-time parameter updates, regularization, warm starting, larger step sizes, line-search when L is not known, mini-batches for vectorized computation, and non-uniform example selection. These details show that SAG can be efficiently implemented for large-scale problems and can be adapted to various problem structures. The paper concludes that SAG is a powerful method for minimizing finite sums of smooth convex functions, with faster convergence rates and better practical performance compared to SG and deterministic gradient methods.