This paper addresses the large-scale matrix factorization problem, specifically focusing on learning a basis set to adapt it to specific data. The authors propose an online optimization algorithm based on stochastic approximations, which is designed to scale well to large datasets with millions of training samples and can be extended to various matrix factorization formulations. The algorithm is shown to converge almost surely to a stationary point of the objective function, and experiments with natural images and genomic data demonstrate its effectiveness in terms of speed and optimization for both small and large datasets. The paper also discusses the algorithm's performance on dictionary learning, non-negative matrix factorization, and sparse principal component analysis, highlighting its versatility and efficiency. Key contributions include the formulation of the dictionary learning problem as the optimization of a smooth nonconvex objective function over a convex set, the development of an iterative online algorithm, and a proof of convergence. The algorithm is further optimized through various techniques, such as handling fixed-size datasets, scaling past data, and using mini-batches, to improve convergence speed and performance.This paper addresses the large-scale matrix factorization problem, specifically focusing on learning a basis set to adapt it to specific data. The authors propose an online optimization algorithm based on stochastic approximations, which is designed to scale well to large datasets with millions of training samples and can be extended to various matrix factorization formulations. The algorithm is shown to converge almost surely to a stationary point of the objective function, and experiments with natural images and genomic data demonstrate its effectiveness in terms of speed and optimization for both small and large datasets. The paper also discusses the algorithm's performance on dictionary learning, non-negative matrix factorization, and sparse principal component analysis, highlighting its versatility and efficiency. Key contributions include the formulation of the dictionary learning problem as the optimization of a smooth nonconvex objective function over a convex set, the development of an iterative online algorithm, and a proof of convergence. The algorithm is further optimized through various techniques, such as handling fixed-size datasets, scaling past data, and using mini-batches, to improve convergence speed and performance.