This paper addresses the issue that instruction-tuned language models (LMs) fail to produce diverse outputs when instructed to generate random samples, even when prompts are carefully designed to encourage randomness. The authors propose a fine-tuning method that encourages LM outputs to be diffuse over valid choices, which generalizes across various tasks and distributions. This method enables large language models to be practical for synthetic dataset generation with minimal human intervention. The authors demonstrate that current LM outputs are highly biased, for example, Llama-2-13B-chat disproportionately favors the number five when asked to pick a number between one and ten, and Mistral-7B-Instruct chooses the name Avery 40 times more often than expected based on U.S. population data. These biases are problematic for tasks requiring diverse outputs, such as synthetic dataset creation. The proposed method, based on distribution matching, encourages LM outputs to align with a desired distribution. It is shown to be effective in tasks like random baby name generation and random number generation, where the model's output distributions are significantly improved. The method also generalizes to new tasks and sample spaces, as demonstrated in experiments where models fine-tuned for one task perform well on unrelated tasks. The authors also show that the method can be applied to synthetic dataset generation, significantly increasing the diversity of generated biographies. For example, the fine-tuned model generates four times as many unique first names, three times as many unique birth places, and 1.5 times as many unique careers as the baseline model, without requiring complex prompt engineering. The method is implemented using parameter-efficient fine-tuning with LoRA, which allows for effective training of large models with minimal memory overhead. The results show that the fine-tuned models produce more diverse outputs, with higher entropy and coverage, and that the method does not significantly affect the general capabilities of the models, as demonstrated by evaluations on MT-Bench. The study highlights the potential of the method for improving diversity in language model generation and reducing the need for human intervention in synthetic dataset creation.This paper addresses the issue that instruction-tuned language models (LMs) fail to produce diverse outputs when instructed to generate random samples, even when prompts are carefully designed to encourage randomness. The authors propose a fine-tuning method that encourages LM outputs to be diffuse over valid choices, which generalizes across various tasks and distributions. This method enables large language models to be practical for synthetic dataset generation with minimal human intervention. The authors demonstrate that current LM outputs are highly biased, for example, Llama-2-13B-chat disproportionately favors the number five when asked to pick a number between one and ten, and Mistral-7B-Instruct chooses the name Avery 40 times more often than expected based on U.S. population data. These biases are problematic for tasks requiring diverse outputs, such as synthetic dataset creation. The proposed method, based on distribution matching, encourages LM outputs to align with a desired distribution. It is shown to be effective in tasks like random baby name generation and random number generation, where the model's output distributions are significantly improved. The method also generalizes to new tasks and sample spaces, as demonstrated in experiments where models fine-tuned for one task perform well on unrelated tasks. The authors also show that the method can be applied to synthetic dataset generation, significantly increasing the diversity of generated biographies. For example, the fine-tuned model generates four times as many unique first names, three times as many unique birth places, and 1.5 times as many unique careers as the baseline model, without requiring complex prompt engineering. The method is implemented using parameter-efficient fine-tuning with LoRA, which allows for effective training of large models with minimal memory overhead. The results show that the fine-tuned models produce more diverse outputs, with higher entropy and coverage, and that the method does not significantly affect the general capabilities of the models, as demonstrated by evaluations on MT-Bench. The study highlights the potential of the method for improving diversity in language model generation and reducing the need for human intervention in synthetic dataset creation.