This paper introduces a novel evaluation framework for Large Language Models (LLMs) using Precision and Recall metrics, originally developed for image generation, to assess the quality and diversity of text generation. The framework allows for a nuanced evaluation of LLMs without requiring aligned corpora. The study evaluates state-of-the-art language models, such as LLAMA-2 and MISTRAL, revealing new insights into their performance on open-ended generation tasks, which are not adequately captured by traditional benchmarks. The findings highlight a trade-off between the quality and diversity of generated samples, particularly when models are fine-tuned on instruction datasets or with human feedback. This work extends the toolkit for distribution-based NLP evaluation, offering insights into the practical capabilities and challenges that current LLMs face in generating diverse and high-quality text. The authors release their code and data. The paper discusses the limitations of traditional benchmarks in evaluating open-ended generation tasks and proposes a new approach using Precision and Recall metrics. These metrics allow for a more detailed assessment of the quality and diversity of generated text. The study shows that these metrics significantly improve the evaluation of LLMs by providing a better understanding of the flaws in text generation. Precision and Recall allow for a clear distinction between sample quality or adequacy and a lack of diversity in model outputs. Empirical results show that these two measures are necessary for a deep comparison of LLMs. For instance, the study finds that fine-tuning models on instruction sets with human feedback significantly improves sample quality, albeit at the expense of sample diversity. The paper also discusses the practical use cases of LLMs, including the evaluation of open-ended generation, the generation of biographies, and creative text generation. The results show that instruction-tuned models are more precise but less diverse than pre-trained models. Larger models are more diverse, and the number of in-context examples affects the precision and recall of models. The study also explores the correlation between Precision and Recall metrics and human evaluation, finding that Precision correlates with quality evaluation and Recall correlates with diversity evaluation. The paper concludes that the proposed metrics provide a valuable tool for evaluating the open-ended generation capabilities of LLMs and contribute to the advancement of generative model evaluation. The study also highlights the ethical considerations of using these metrics, emphasizing the need for well-designed reference datasets to avoid biases.This paper introduces a novel evaluation framework for Large Language Models (LLMs) using Precision and Recall metrics, originally developed for image generation, to assess the quality and diversity of text generation. The framework allows for a nuanced evaluation of LLMs without requiring aligned corpora. The study evaluates state-of-the-art language models, such as LLAMA-2 and MISTRAL, revealing new insights into their performance on open-ended generation tasks, which are not adequately captured by traditional benchmarks. The findings highlight a trade-off between the quality and diversity of generated samples, particularly when models are fine-tuned on instruction datasets or with human feedback. This work extends the toolkit for distribution-based NLP evaluation, offering insights into the practical capabilities and challenges that current LLMs face in generating diverse and high-quality text. The authors release their code and data. The paper discusses the limitations of traditional benchmarks in evaluating open-ended generation tasks and proposes a new approach using Precision and Recall metrics. These metrics allow for a more detailed assessment of the quality and diversity of generated text. The study shows that these metrics significantly improve the evaluation of LLMs by providing a better understanding of the flaws in text generation. Precision and Recall allow for a clear distinction between sample quality or adequacy and a lack of diversity in model outputs. Empirical results show that these two measures are necessary for a deep comparison of LLMs. For instance, the study finds that fine-tuning models on instruction sets with human feedback significantly improves sample quality, albeit at the expense of sample diversity. The paper also discusses the practical use cases of LLMs, including the evaluation of open-ended generation, the generation of biographies, and creative text generation. The results show that instruction-tuned models are more precise but less diverse than pre-trained models. Larger models are more diverse, and the number of in-context examples affects the precision and recall of models. The study also explores the correlation between Precision and Recall metrics and human evaluation, finding that Precision correlates with quality evaluation and Recall correlates with diversity evaluation. The paper concludes that the proposed metrics provide a valuable tool for evaluating the open-ended generation capabilities of LLMs and contribute to the advancement of generative model evaluation. The study also highlights the ethical considerations of using these metrics, emphasizing the need for well-designed reference datasets to avoid biases.