This paper introduces Alignment from Demonstrations (AfD), a novel approach for aligning Large Language Models (LLMs) using high-quality demonstration data, which addresses the limitations of preference-based alignment methods. AfD leverages sequential decision-making frameworks and inverse reinforcement learning principles to overcome challenges such as noisy labels, high annotation costs, and privacy concerns. The method introduces divergence minimization objectives for AfD, enabling efficient alignment without the need for continuous human feedback or external annotators. The paper demonstrates that AfD can be framed within a Markov Decision Process (MDP) and connects it with inverse reinforcement learning to enhance understanding of potential solutions. The authors propose a computationally efficient algorithm for AfD that extrapolates over a tailored reward model, validated through experiments on the Harmless and Helpful tasks of the Anthropic HH-RLHF dataset. The results show that AfD achieves strong empirical performance while maintaining simplicity. The paper also highlights the challenges of reward hacking in AfD and proposes an easy-to-implement algorithm to address this issue. AfD is shown to be a viable and efficient alternative to reinforcement learning from human feedback (RLHF), offering a way to align LLMs without the need for preference-based data. The method is particularly effective in tasks where response variability is high, such as the Helpful task, where SFT alone is insufficient. The paper also discusses the benefits of using demonstration data, which avoids the noise, cost, and assumptions inherent in preference-based methods, and preserves data privacy. The study provides theoretical insights into the behavior of various approaches, including mass-covering and mode-seeking behaviors, and demonstrates the effectiveness of AfD in achieving superior alignment performance. The results validate the potential of AfD as a safe and effective alternative to RLHF, paving the way for safer and more reliable deployment of LLMs in various applications.This paper introduces Alignment from Demonstrations (AfD), a novel approach for aligning Large Language Models (LLMs) using high-quality demonstration data, which addresses the limitations of preference-based alignment methods. AfD leverages sequential decision-making frameworks and inverse reinforcement learning principles to overcome challenges such as noisy labels, high annotation costs, and privacy concerns. The method introduces divergence minimization objectives for AfD, enabling efficient alignment without the need for continuous human feedback or external annotators. The paper demonstrates that AfD can be framed within a Markov Decision Process (MDP) and connects it with inverse reinforcement learning to enhance understanding of potential solutions. The authors propose a computationally efficient algorithm for AfD that extrapolates over a tailored reward model, validated through experiments on the Harmless and Helpful tasks of the Anthropic HH-RLHF dataset. The results show that AfD achieves strong empirical performance while maintaining simplicity. The paper also highlights the challenges of reward hacking in AfD and proposes an easy-to-implement algorithm to address this issue. AfD is shown to be a viable and efficient alternative to reinforcement learning from human feedback (RLHF), offering a way to align LLMs without the need for preference-based data. The method is particularly effective in tasks where response variability is high, such as the Helpful task, where SFT alone is insufficient. The paper also discusses the benefits of using demonstration data, which avoids the noise, cost, and assumptions inherent in preference-based methods, and preserves data privacy. The study provides theoretical insights into the behavior of various approaches, including mass-covering and mode-seeking behaviors, and demonstrates the effectiveness of AfD in achieving superior alignment performance. The results validate the potential of AfD as a safe and effective alternative to RLHF, paving the way for safer and more reliable deployment of LLMs in various applications.