This paper provides a comprehensive survey of recent advances in mathematical optimization theory and algorithms for wireless communication system design. It discusses various optimization problems arising in wireless communication system design, including nonconvex optimization, global optimization, integer programming, distributed optimization, and learning-based optimization. The paper highlights the importance of mathematical optimization in wireless communication system design and the challenges posed by the increasing complexity of optimization problems in modern wireless systems. It also identifies several open research challenges and outlines future research directions. The paper reviews recent advances in structured nonconvex optimization, including fractional programming, sparse optimization, proximal gradient algorithms, penalty methods, and duality-based algorithms. It also discusses the application of these techniques in various wireless communication scenarios, such as beamforming, power control, resource allocation, and scheduling. The paper emphasizes the need for developing suitable algorithms that can exploit the underlying problem structure to efficiently solve optimization problems in wireless communication systems. It concludes by highlighting the importance of cross-fertilization between mathematical optimization and wireless communications, and the potential for new optimization techniques to drive advancements in wireless communication technologies.This paper provides a comprehensive survey of recent advances in mathematical optimization theory and algorithms for wireless communication system design. It discusses various optimization problems arising in wireless communication system design, including nonconvex optimization, global optimization, integer programming, distributed optimization, and learning-based optimization. The paper highlights the importance of mathematical optimization in wireless communication system design and the challenges posed by the increasing complexity of optimization problems in modern wireless systems. It also identifies several open research challenges and outlines future research directions. The paper reviews recent advances in structured nonconvex optimization, including fractional programming, sparse optimization, proximal gradient algorithms, penalty methods, and duality-based algorithms. It also discusses the application of these techniques in various wireless communication scenarios, such as beamforming, power control, resource allocation, and scheduling. The paper emphasizes the need for developing suitable algorithms that can exploit the underlying problem structure to efficiently solve optimization problems in wireless communication systems. It concludes by highlighting the importance of cross-fertilization between mathematical optimization and wireless communications, and the potential for new optimization techniques to drive advancements in wireless communication technologies.