NeMo-Aligner is a scalable toolkit designed for efficient model alignment, particularly for large language models (LLMs). It addresses the challenge of aligning LLMs with human values and preferences, which is crucial for making these models helpful and safe. The toolkit supports various alignment techniques, including Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), SteerLM, and Self-Play Fine-Tuning (SPIN), and can run these techniques in a Parameter Efficient Fine-Tuning (PEFT) setting. NeMo-Aligner is optimized for scalability, leveraging 3D parallelism, distributed training, and TensorRT-LLM optimizations to handle large models trained on hundreds of GPUs. The toolkit is open-sourced under the Apache 2.0 license, encouraging community contributions. It demonstrates practical large-scale RLHF training with a 70B Llama 2 model, achieving a high MT-Bench score. NeMo-Aligner also supports other alignment techniques like DPO variants, LoRA for parameter-efficient fine-tuning, and SPIN, showcasing its versatility and efficiency in model alignment research.NeMo-Aligner is a scalable toolkit designed for efficient model alignment, particularly for large language models (LLMs). It addresses the challenge of aligning LLMs with human values and preferences, which is crucial for making these models helpful and safe. The toolkit supports various alignment techniques, including Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), SteerLM, and Self-Play Fine-Tuning (SPIN), and can run these techniques in a Parameter Efficient Fine-Tuning (PEFT) setting. NeMo-Aligner is optimized for scalability, leveraging 3D parallelism, distributed training, and TensorRT-LLM optimizations to handle large models trained on hundreds of GPUs. The toolkit is open-sourced under the Apache 2.0 license, encouraging community contributions. It demonstrates practical large-scale RLHF training with a 70B Llama 2 model, achieving a high MT-Bench score. NeMo-Aligner also supports other alignment techniques like DPO variants, LoRA for parameter-efficient fine-tuning, and SPIN, showcasing its versatility and efficiency in model alignment research.