This paper presents a general framework for solving linear inverse problems where the number of available measurements is insufficient to uniquely determine the model. The framework converts notions of simplicity into convex penalty functions, leading to convex optimization solutions. The models considered are those formed as the sum of a few atoms from an atomic set, such as sparse vectors, low-rank matrices, sums of permutation matrices, low-rank tensors, orthogonal matrices, and atomic measures. The convex programming formulation minimizes the atomic norm, which is induced by the convex hull of the atomic set. The facial structure of the atomic norm ball is analyzed to provide sharp estimates on the number of generic measurements required for exact and robust recovery. These estimates are based on computing the Gaussian widths of tangent cones to the atomic norm ball. When the atomic set has algebraic structure, the resulting optimization problems can be solved or approximated via semidefinite programming. The paper also discusses the connection between algebraic structure and semidefinite representability, and provides a hierarchy of semidefinite relaxations using theta bodies. The quality of these relaxations affects the number of measurements required for recovery, and this tradeoff is characterized through examples. The work extends the catalog of simple models that can be recovered from limited linear information via tractable convex programming.This paper presents a general framework for solving linear inverse problems where the number of available measurements is insufficient to uniquely determine the model. The framework converts notions of simplicity into convex penalty functions, leading to convex optimization solutions. The models considered are those formed as the sum of a few atoms from an atomic set, such as sparse vectors, low-rank matrices, sums of permutation matrices, low-rank tensors, orthogonal matrices, and atomic measures. The convex programming formulation minimizes the atomic norm, which is induced by the convex hull of the atomic set. The facial structure of the atomic norm ball is analyzed to provide sharp estimates on the number of generic measurements required for exact and robust recovery. These estimates are based on computing the Gaussian widths of tangent cones to the atomic norm ball. When the atomic set has algebraic structure, the resulting optimization problems can be solved or approximated via semidefinite programming. The paper also discusses the connection between algebraic structure and semidefinite representability, and provides a hierarchy of semidefinite relaxations using theta bodies. The quality of these relaxations affects the number of measurements required for recovery, and this tradeoff is characterized through examples. The work extends the catalog of simple models that can be recovered from limited linear information via tractable convex programming.