This paper provides a comprehensive survey of recent advances in mathematical optimization theory and algorithms for wireless communication system design. It begins by illustrating common features of mathematical optimization problems arising in wireless communication system design, discussing various scenarios and use cases, and their associated mathematical structures. The paper then reviews recent developments in optimization techniques, including nonconvex optimization, global optimization, integer programming, distributed optimization, and learning-based optimization. The key to successful solution of these problems lies in carefully choosing or developing suitable algorithms that can exploit the underlying problem structure. The paper concludes by identifying several open research challenges and outlining future research directions. The evolution of wireless communication systems from 1G to 6G has driven significant innovations in both physical and networking technologies, leading to more stringent key performance indicators (KPIs). These advancements have transformed the nature of underlying mathematical optimization problems, necessitating new optimization theory and algorithms. The paper aims to guide the choice and development of suitable algorithms for solving structured optimization problems and promote the cross-fertilization of ideas between mathematical optimization and wireless communications.This paper provides a comprehensive survey of recent advances in mathematical optimization theory and algorithms for wireless communication system design. It begins by illustrating common features of mathematical optimization problems arising in wireless communication system design, discussing various scenarios and use cases, and their associated mathematical structures. The paper then reviews recent developments in optimization techniques, including nonconvex optimization, global optimization, integer programming, distributed optimization, and learning-based optimization. The key to successful solution of these problems lies in carefully choosing or developing suitable algorithms that can exploit the underlying problem structure. The paper concludes by identifying several open research challenges and outlining future research directions. The evolution of wireless communication systems from 1G to 6G has driven significant innovations in both physical and networking technologies, leading to more stringent key performance indicators (KPIs). These advancements have transformed the nature of underlying mathematical optimization problems, necessitating new optimization theory and algorithms. The paper aims to guide the choice and development of suitable algorithms for solving structured optimization problems and promote the cross-fertilization of ideas between mathematical optimization and wireless communications.