23 Jan 2024 | Yunpu Zhao, Rui Zhang, Wenyi Li, Di Huang, Jiaming Guo, Shaohui Peng, Yifan Hao, Yuanbo Wen, Xing Hu, Zidong Du, Qi Guo, Ling Li and Yunji Chen
This paper presents a framework for assessing the creativity of large language models (LLMs). The framework is based on the Torrance Tests of Creative Thinking (TTCT), a widely used tool for evaluating human creativity. The study evaluates the creative performance of various LLMs across seven tasks, emphasizing four criteria: Fluency, Flexibility, Originality, and Elaboration. A dataset of 700 questions was constructed, and an LLM-based evaluation method was developed. The study also analyzes LLM responses to diverse prompts and role-play situations, revealing that LLMs primarily fall short in originality but excel in elaboration. The use of prompts and role-play settings significantly influences creativity. Additionally, collaboration among multiple LLMs can enhance originality. The study also explores the relationship between LLMs and human creativity, finding a consensus in personality traits such as emotional intelligence, empathy, self-efficacy, and others. The findings highlight the significant impact of LLM design on creativity and bridge artificial intelligence and human creativity, offering insights into LLMs' creativity and potential applications. The study demonstrates that LLMs' creativity is significantly influenced by model architecture, prompt type, and system prompts. The results show that different roles can influence creativity, with scientists showing the highest level of creativity. Collaboration among LLMs can enhance creativity, particularly in originality. The study also finds a correlation between LLMs' personality traits and creativity, with emotional intelligence, empathy, conscientiousness, extraversion, and neuroticism showing significant positive correlations with creativity, while agreeableness shows a significant negative correlation. The study concludes that LLMs' creativity is influenced by various factors, and further research is needed to explore the creativity of multimodal models and other types of generative models.This paper presents a framework for assessing the creativity of large language models (LLMs). The framework is based on the Torrance Tests of Creative Thinking (TTCT), a widely used tool for evaluating human creativity. The study evaluates the creative performance of various LLMs across seven tasks, emphasizing four criteria: Fluency, Flexibility, Originality, and Elaboration. A dataset of 700 questions was constructed, and an LLM-based evaluation method was developed. The study also analyzes LLM responses to diverse prompts and role-play situations, revealing that LLMs primarily fall short in originality but excel in elaboration. The use of prompts and role-play settings significantly influences creativity. Additionally, collaboration among multiple LLMs can enhance originality. The study also explores the relationship between LLMs and human creativity, finding a consensus in personality traits such as emotional intelligence, empathy, self-efficacy, and others. The findings highlight the significant impact of LLM design on creativity and bridge artificial intelligence and human creativity, offering insights into LLMs' creativity and potential applications. The study demonstrates that LLMs' creativity is significantly influenced by model architecture, prompt type, and system prompts. The results show that different roles can influence creativity, with scientists showing the highest level of creativity. Collaboration among LLMs can enhance creativity, particularly in originality. The study also finds a correlation between LLMs' personality traits and creativity, with emotional intelligence, empathy, conscientiousness, extraversion, and neuroticism showing significant positive correlations with creativity, while agreeableness shows a significant negative correlation. The study concludes that LLMs' creativity is influenced by various factors, and further research is needed to explore the creativity of multimodal models and other types of generative models.