NeMo-Aligner is a scalable toolkit for efficient model alignment, designed to handle large language models (LLMs) with tens or hundreds of billions of parameters. It supports major alignment paradigms such as Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), SteerLM, and Self-Play Fine-Tuning (SPIN), and is optimized for Parameter Efficient Fine-Tuning (PEFT). The toolkit is open-sourced under the Apache 2.0 license and allows for easy extension to support new alignment techniques. The toolkit addresses scalability challenges by leveraging 3D parallelism, distributed Proximal Policy Optimization (PPO) training, and TensorRT-LLM for inference optimizations. It enables efficient training of large models using hundreds of GPUs, reducing research iteration time. NeMo-Aligner optimizes popular alignment techniques including Supervised Finetuning (SFT), PPO-based RLHF, DPO, SteerLM, and SPIN. It provides a flexible framework for training and aligning models, with support for various alignment methods and efficient scaling. The toolkit is designed for extensibility, allowing integration of new alignment techniques with minimal effort. It supports a wide range of alignment methods, including RLHF, DPO, SteerLM, and SPIN, and is optimized for performance and scalability. NeMo-Aligner is tested on large models such as Llama 2 70B and demonstrates significant improvements in performance and efficiency. The framework is also compatible with parameter-efficient fine-tuning techniques like LoRA, enabling efficient training on compute-limited settings. NeMo-Aligner is designed to be scalable and efficient, with optimizations for both training and inference stages. It supports distributed training and inference, with optimizations for PPO training and inference. The toolkit is open-sourced and provides a flexible framework for researchers and practitioners to experiment with LLM alignment. It is intended to make model alignment more efficient and accessible, enabling the development of safer and more helpful large language models.NeMo-Aligner is a scalable toolkit for efficient model alignment, designed to handle large language models (LLMs) with tens or hundreds of billions of parameters. It supports major alignment paradigms such as Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), SteerLM, and Self-Play Fine-Tuning (SPIN), and is optimized for Parameter Efficient Fine-Tuning (PEFT). The toolkit is open-sourced under the Apache 2.0 license and allows for easy extension to support new alignment techniques. The toolkit addresses scalability challenges by leveraging 3D parallelism, distributed Proximal Policy Optimization (PPO) training, and TensorRT-LLM for inference optimizations. It enables efficient training of large models using hundreds of GPUs, reducing research iteration time. NeMo-Aligner optimizes popular alignment techniques including Supervised Finetuning (SFT), PPO-based RLHF, DPO, SteerLM, and SPIN. It provides a flexible framework for training and aligning models, with support for various alignment methods and efficient scaling. The toolkit is designed for extensibility, allowing integration of new alignment techniques with minimal effort. It supports a wide range of alignment methods, including RLHF, DPO, SteerLM, and SPIN, and is optimized for performance and scalability. NeMo-Aligner is tested on large models such as Llama 2 70B and demonstrates significant improvements in performance and efficiency. The framework is also compatible with parameter-efficient fine-tuning techniques like LoRA, enabling efficient training on compute-limited settings. NeMo-Aligner is designed to be scalable and efficient, with optimizations for both training and inference stages. It supports distributed training and inference, with optimizations for PPO training and inference. The toolkit is open-sourced and provides a flexible framework for researchers and practitioners to experiment with LLM alignment. It is intended to make model alignment more efficient and accessible, enabling the development of safer and more helpful large language models.