ControlVAR is a novel framework for controllable visual autoregressive modeling, designed to address the challenges of expensive computational costs, high inference latency, and difficulties in integrating with large language models (LLMs) associated with diffusion models (DMs). Unlike traditional conditional models that learn the conditional distribution, ControlVAR jointly models the distribution of images and pixel-level controls during training and imposes conditional controls during testing. The framework adopts a next-scale AR prediction paradigm and unifies control and image representations to enhance joint modeling. A teacher-forcing guidance strategy is introduced to facilitate controllable generation with joint modeling. Extensive experiments demonstrate the superior efficacy and flexibility of ControlVAR across various conditional generation tasks, outperforming popular conditional DMs such as ControlNet and T2I-Adapter. ControlVAR supports multiple tasks, including joint control-image generation, control/image completion, control-to-image generation, and image-to-control prediction, showcasing its versatility and robustness.ControlVAR is a novel framework for controllable visual autoregressive modeling, designed to address the challenges of expensive computational costs, high inference latency, and difficulties in integrating with large language models (LLMs) associated with diffusion models (DMs). Unlike traditional conditional models that learn the conditional distribution, ControlVAR jointly models the distribution of images and pixel-level controls during training and imposes conditional controls during testing. The framework adopts a next-scale AR prediction paradigm and unifies control and image representations to enhance joint modeling. A teacher-forcing guidance strategy is introduced to facilitate controllable generation with joint modeling. Extensive experiments demonstrate the superior efficacy and flexibility of ControlVAR across various conditional generation tasks, outperforming popular conditional DMs such as ControlNet and T2I-Adapter. ControlVAR supports multiple tasks, including joint control-image generation, control/image completion, control-to-image generation, and image-to-control prediction, showcasing its versatility and robustness.