This paper presents a comprehensive overview of optimization techniques for sparsity-inducing penalties in machine learning. The goal is to provide a general perspective on optimization tools and methods that are relevant for sparse estimation problems. The paper covers various approaches, including proximal methods, block-coordinate descent, reweighted $ \ell_2 $-penalized techniques, working-set and homotopy methods, as well as non-convex formulations and extensions. It also provides an extensive set of experiments to compare various algorithms from a computational point of view. The paper begins with an introduction to the principles of sparsity-inducing norms and their applications in machine learning. It discusses the use of sparsity-inducing norms for variable selection, structured sparsity, and multiple kernel learning. The paper then presents a detailed discussion of various optimization techniques, including proximal methods, block-coordinate descent algorithms, reweighted-$ \ell_2 $ algorithms, working-set and homotopy methods, and non-convex optimization approaches. It also covers the use of sparsity-inducing norms in Bayesian methods and sparse matrix factorization. The paper also discusses the computational aspects of these methods, including speed benchmarks for Lasso, group-sparsity for multi-task learning, and structured sparsity. It provides a detailed analysis of the properties of sparsity-inducing norms, including their geometric interpretation and their role in promoting sparsity in the solution. The paper also discusses the use of dual norms and Fenchel duality in the analysis of sparsity-inducing regularizations. The paper concludes with a discussion of the broader implications of sparsity-inducing norms in machine learning, including their applications in signal processing, computer vision, text processing, bioinformatics, and audio processing. It also discusses the potential for extending these methods to non-convex formulations and the importance of computational efficiency in the design of sparse estimation algorithms. The paper provides a comprehensive overview of the key concepts and techniques in sparsity-inducing optimization, making it a valuable resource for researchers and practitioners in the field of machine learning.This paper presents a comprehensive overview of optimization techniques for sparsity-inducing penalties in machine learning. The goal is to provide a general perspective on optimization tools and methods that are relevant for sparse estimation problems. The paper covers various approaches, including proximal methods, block-coordinate descent, reweighted $ \ell_2 $-penalized techniques, working-set and homotopy methods, as well as non-convex formulations and extensions. It also provides an extensive set of experiments to compare various algorithms from a computational point of view. The paper begins with an introduction to the principles of sparsity-inducing norms and their applications in machine learning. It discusses the use of sparsity-inducing norms for variable selection, structured sparsity, and multiple kernel learning. The paper then presents a detailed discussion of various optimization techniques, including proximal methods, block-coordinate descent algorithms, reweighted-$ \ell_2 $ algorithms, working-set and homotopy methods, and non-convex optimization approaches. It also covers the use of sparsity-inducing norms in Bayesian methods and sparse matrix factorization. The paper also discusses the computational aspects of these methods, including speed benchmarks for Lasso, group-sparsity for multi-task learning, and structured sparsity. It provides a detailed analysis of the properties of sparsity-inducing norms, including their geometric interpretation and their role in promoting sparsity in the solution. The paper also discusses the use of dual norms and Fenchel duality in the analysis of sparsity-inducing regularizations. The paper concludes with a discussion of the broader implications of sparsity-inducing norms in machine learning, including their applications in signal processing, computer vision, text processing, bioinformatics, and audio processing. It also discusses the potential for extending these methods to non-convex formulations and the importance of computational efficiency in the design of sparse estimation algorithms. The paper provides a comprehensive overview of the key concepts and techniques in sparsity-inducing optimization, making it a valuable resource for researchers and practitioners in the field of machine learning.