This paper explores the impact of data diversity on fine-tuning large language models (LLMs) for human alignment, aiming to prevent the generation of misleading or toxic content. The authors investigate two main strategies: increasing the number of prompts or responses for each prompt. Through a series of experiments, they find that increasing the number of responses is more effective in improving LLM performance compared to expanding the number of prompts. They also propose a new metric to measure prompt diversity, which shows a linear correlation with the final performance of fine-tuned LLMs. Additionally, the paper introduces a method for data augmentation that enhances prompt diversity, leading to better performance. The findings highlight the importance of balancing the allocation of human annotations between prompts and responses to achieve optimal human alignment in LLM fine-tuning.This paper explores the impact of data diversity on fine-tuning large language models (LLMs) for human alignment, aiming to prevent the generation of misleading or toxic content. The authors investigate two main strategies: increasing the number of prompts or responses for each prompt. Through a series of experiments, they find that increasing the number of responses is more effective in improving LLM performance compared to expanding the number of prompts. They also propose a new metric to measure prompt diversity, which shows a linear correlation with the final performance of fine-tuned LLMs. Additionally, the paper introduces a method for data augmentation that enhances prompt diversity, leading to better performance. The findings highlight the importance of balancing the allocation of human annotations between prompts and responses to achieve optimal human alignment in LLM fine-tuning.