Limited ability of LLMs to simulate human psychological behaviours: a psychometric analysis

Limited ability of LLMs to simulate human psychological behaviours: a psychometric analysis

12 May 2024 | Nikolay B Petrov, Gregory Serapio-García, Jason Rentfrow
The study investigates the ability of large language models (LLMs) to simulate human psychological behaviors using psychometric analysis. Specifically, it examines whether LLMs can be prompted to assume different personas and respond to standardized measures of personality constructs. The research focuses on two models, GPT-3.5 and GPT-4, and uses two types of persona descriptions: generic and specific (based on demographic data from a large-scale human dataset). The findings indicate that while GPT-4 shows promising psychometric properties when using generic persona descriptions, both models perform poorly when using specific demographic profiles. The study concludes that current LLMs are not yet capable of accurately simulating individual-level human behaviors, particularly when using silicon personas. The results highlight the limitations of LLMs in representing underlying latent traits and suggest that further research is needed to improve their psychometric properties.The study investigates the ability of large language models (LLMs) to simulate human psychological behaviors using psychometric analysis. Specifically, it examines whether LLMs can be prompted to assume different personas and respond to standardized measures of personality constructs. The research focuses on two models, GPT-3.5 and GPT-4, and uses two types of persona descriptions: generic and specific (based on demographic data from a large-scale human dataset). The findings indicate that while GPT-4 shows promising psychometric properties when using generic persona descriptions, both models perform poorly when using specific demographic profiles. The study concludes that current LLMs are not yet capable of accurately simulating individual-level human behaviors, particularly when using silicon personas. The results highlight the limitations of LLMs in representing underlying latent traits and suggest that further research is needed to improve their psychometric properties.
Reach us at info@study.space
[slides] Limited Ability of LLMs to Simulate Human Psychological Behaviours%3A a Psychometric Analysis | StudySpace