**Abstract:** The paper introduces *Aligner*, a novel and efficient alignment paradigm designed to correct the responses of large language models (LLMs) using a small, model-agnostic module. *Aligner* learns to correct the residuals between preferred and dispreferred answers, enhancing the helpfulness, harmlessness, and honesty (3H) dimensions of LLMs. The method is lightweight, requiring only a single training session and making it suitable for rapid iteration and API-based models. Experiments show that *Aligner-7B* improves helpfulness by 68.9% and harmlessness by 23.8% across 11 different LLMs, outperforming other alignment methods in terms of resource efficiency and performance. **Introduction:** The alignment of LLMs with human intentions and values is crucial but challenging due to the complexity of current methods and the need for rapid iteration. *Aligner* addresses these issues by focusing on copy and correction operations, simplifying the alignment process. It is trained on a preference dataset to learn correctional residuals and can be applied to various LLMs without significant changes to the upstream model. The method is efficient, requiring minimal computational resources and offering a plug-and-play solution. **Aligner:** *Aligner* is a conditional seq2seq model that redistributes initial answers from the upstream LLM to more helpful and harmless responses. It is trained to optimize the upper bound of the supervised fine-tuning (SFT) objective, ensuring effective learning of correctional residuals. The training process involves a warm-up step to learn identity mapping, followed by residual Q-A-C learning. **Experiments:** *Aligner* is evaluated on five datasets using the 3H standard, demonstrating significant improvements in helpfulness and harmlessness across various models. It outperforms other alignment methods in terms of resource efficiency and performance. Ablation studies show that the warm-up step and the correction paradigm are crucial for effective alignment. **Multi-round RLHF Training:** *Aligner* can enhance the performance of LLMs in multi-round reinforcement learning from human feedback (RLHF) and data optimization (DPO) pipelines by providing synthetic preference datasets. This approach mitigates reward collapse and improves safety and helpfulness. **Conclusion:** *Aligner* is an efficient and lightweight alignment method that significantly enhances the performance of LLMs in terms of helpfulness, harmlessness, and honesty. It offers a plug-and-play solution suitable for various models and deployment scenarios.**Abstract:** The paper introduces *Aligner*, a novel and efficient alignment paradigm designed to correct the responses of large language models (LLMs) using a small, model-agnostic module. *Aligner* learns to correct the residuals between preferred and dispreferred answers, enhancing the helpfulness, harmlessness, and honesty (3H) dimensions of LLMs. The method is lightweight, requiring only a single training session and making it suitable for rapid iteration and API-based models. Experiments show that *Aligner-7B* improves helpfulness by 68.9% and harmlessness by 23.8% across 11 different LLMs, outperforming other alignment methods in terms of resource efficiency and performance. **Introduction:** The alignment of LLMs with human intentions and values is crucial but challenging due to the complexity of current methods and the need for rapid iteration. *Aligner* addresses these issues by focusing on copy and correction operations, simplifying the alignment process. It is trained on a preference dataset to learn correctional residuals and can be applied to various LLMs without significant changes to the upstream model. The method is efficient, requiring minimal computational resources and offering a plug-and-play solution. **Aligner:** *Aligner* is a conditional seq2seq model that redistributes initial answers from the upstream LLM to more helpful and harmless responses. It is trained to optimize the upper bound of the supervised fine-tuning (SFT) objective, ensuring effective learning of correctional residuals. The training process involves a warm-up step to learn identity mapping, followed by residual Q-A-C learning. **Experiments:** *Aligner* is evaluated on five datasets using the 3H standard, demonstrating significant improvements in helpfulness and harmlessness across various models. It outperforms other alignment methods in terms of resource efficiency and performance. Ablation studies show that the warm-up step and the correction paradigm are crucial for effective alignment. **Multi-round RLHF Training:** *Aligner* can enhance the performance of LLMs in multi-round reinforcement learning from human feedback (RLHF) and data optimization (DPO) pipelines by providing synthetic preference datasets. This approach mitigates reward collapse and improves safety and helpfulness. **Conclusion:** *Aligner* is an efficient and lightweight alignment method that significantly enhances the performance of LLMs in terms of helpfulness, harmlessness, and honesty. It offers a plug-and-play solution suitable for various models and deployment scenarios.