Is Temperature the Creativity Parameter of Large Language Models?

Is Temperature the Creativity Parameter of Large Language Models?

1 May 2024 | Max Peeperkorn, Tom Kouwenhoven, Dan Brown, Anna Jordanous
The paper investigates whether temperature, a hyperparameter in large language models (LLMs), is the creativity parameter. Using a narrative generation task with a fixed context, model, and prompt, the authors conduct an empirical analysis of LLM outputs for different temperature values, focusing on four necessary conditions for creativity: novelty, typicality, cohesion, and coherence. They find that temperature has a weak correlation with novelty and a moderate correlation with incoherence, but no significant relationship with cohesion or typicality. The influence of temperature on creativity is more nuanced and weak than suggested by the "creativity parameter" claim. The study concludes that LLMs generate slightly more novel outputs as temperatures increase, but the overall impact is limited. The authors discuss the limitations of their approach and suggest future directions, including the need for more controlled creativity experiments and the development of advanced decoding strategies to enhance LLM creativity.The paper investigates whether temperature, a hyperparameter in large language models (LLMs), is the creativity parameter. Using a narrative generation task with a fixed context, model, and prompt, the authors conduct an empirical analysis of LLM outputs for different temperature values, focusing on four necessary conditions for creativity: novelty, typicality, cohesion, and coherence. They find that temperature has a weak correlation with novelty and a moderate correlation with incoherence, but no significant relationship with cohesion or typicality. The influence of temperature on creativity is more nuanced and weak than suggested by the "creativity parameter" claim. The study concludes that LLMs generate slightly more novel outputs as temperatures increase, but the overall impact is limited. The authors discuss the limitations of their approach and suggest future directions, including the need for more controlled creativity experiments and the development of advanced decoding strategies to enhance LLM creativity.
Reach us at info@study.space
[slides and audio] Is Temperature the Creativity Parameter of Large Language Models%3F