FLAME (Factuality-Aware Alignment) is a method to enhance the factual accuracy of large language models (LLMs) during alignment. The conventional alignment process, which includes supervised fine-tuning (SFT) and reinforcement learning (RL), often leads to hallucination, or the generation of false facts. FLAME addresses this by incorporating factuality-aware SFT and RL through direct preference optimization. The method identifies factors that contribute to hallucination in both SFT and RL, such as training on human-labeled data that may contain novel information and reward functions that encourage longer, more detailed responses. FLAME improves factual accuracy by using factuality-aware training data and separate rewards for factuality and instruction following. Experiments show that FLAME significantly improves factual accuracy without compromising instruction-following capability. The method is evaluated on Alpaca Eval and Biography, with results showing a significant increase in FActScore compared to standard alignment. FLAME also demonstrates effectiveness in reducing hallucination and improving factual accuracy in various tasks. The study highlights the importance of factuality in LLM alignment and provides a comprehensive approach to improving both factuality and instruction-following ability.FLAME (Factuality-Aware Alignment) is a method to enhance the factual accuracy of large language models (LLMs) during alignment. The conventional alignment process, which includes supervised fine-tuning (SFT) and reinforcement learning (RL), often leads to hallucination, or the generation of false facts. FLAME addresses this by incorporating factuality-aware SFT and RL through direct preference optimization. The method identifies factors that contribute to hallucination in both SFT and RL, such as training on human-labeled data that may contain novel information and reward functions that encourage longer, more detailed responses. FLAME improves factual accuracy by using factuality-aware training data and separate rewards for factuality and instruction following. Experiments show that FLAME significantly improves factual accuracy without compromising instruction-following capability. The method is evaluated on Alpaca Eval and Biography, with results showing a significant increase in FActScore compared to standard alignment. FLAME also demonstrates effectiveness in reducing hallucination and improving factual accuracy in various tasks. The study highlights the importance of factuality in LLM alignment and provides a comprehensive approach to improving both factuality and instruction-following ability.