This paper evaluates the performance of alignment methods, including Direct Preference Optimization (DPO), Informational Preference Optimization (IPO), KTO, and CPO, across various tasks such as dialogue systems, reasoning, mathematical problem-solving, question answering, truthfulness, and multi-task understanding. The evaluation spans 13 benchmarks, including MT-Bench, Big Bench, and Open LLM Leaderboard. The study investigates three scenarios: fine-tuning a Supervised Fine-Tuning (SFT) model, fine-tuning a pre-trained model, and fine-tuning an instruction-tuned model. Key findings include: 1. **Performance with Smaller Training Data**: Alignment methods achieve optimal performance with smaller training data subsets. 2. **Limited Effectiveness in Reasoning Tasks**: These methods show limited effectiveness in reasoning tasks but significantly impact mathematical problem-solving. 3. **Impact of Instruction-Tuned Models**: Instruction-tuned models notably influence truthfulness. 4. **Superiority of KTO**: KTO outperforms other alignment methods in most benchmarks, except for multi-task understanding. 5. **SFT's Resilience**: SFT remains superior in multitask understanding and comparable in reasoning tasks. 6. **Efficiency Without SFT**: KTO and CPO can bypass the SFT phase and achieve comparable performance on MT-Bench. The study highlights the importance of considering the trade-offs between SFT and RL-free methods and the impact of training data volume on performance. The findings have implications for further research on developing more robust models to address alignment challenges.This paper evaluates the performance of alignment methods, including Direct Preference Optimization (DPO), Informational Preference Optimization (IPO), KTO, and CPO, across various tasks such as dialogue systems, reasoning, mathematical problem-solving, question answering, truthfulness, and multi-task understanding. The evaluation spans 13 benchmarks, including MT-Bench, Big Bench, and Open LLM Leaderboard. The study investigates three scenarios: fine-tuning a Supervised Fine-Tuning (SFT) model, fine-tuning a pre-trained model, and fine-tuning an instruction-tuned model. Key findings include: 1. **Performance with Smaller Training Data**: Alignment methods achieve optimal performance with smaller training data subsets. 2. **Limited Effectiveness in Reasoning Tasks**: These methods show limited effectiveness in reasoning tasks but significantly impact mathematical problem-solving. 3. **Impact of Instruction-Tuned Models**: Instruction-tuned models notably influence truthfulness. 4. **Superiority of KTO**: KTO outperforms other alignment methods in most benchmarks, except for multi-task understanding. 5. **SFT's Resilience**: SFT remains superior in multitask understanding and comparable in reasoning tasks. 6. **Efficiency Without SFT**: KTO and CPO can bypass the SFT phase and achieve comparable performance on MT-Bench. The study highlights the importance of considering the trade-offs between SFT and RL-free methods and the impact of training data volume on performance. The findings have implications for further research on developing more robust models to address alignment challenges.