Large Language Models (LLMs) often assume that people are more rational than they actually are, according to a study. The research compared LLMs' predictions and simulations of human decisions with a large dataset of human choices. It found that LLMs, including GPT-4o, GPT-4-Turbo, Llama-3-8B, Llama-3-70B, and Claude 3 Opus, tend to align more closely with a rational decision-making model (expected value theory) than with actual human behavior. This suggests that LLMs may not accurately represent how people make decisions, especially in risky choices. The study used two psychological paradigms: a risky choice task where participants choose between gambles, and an inference task where people infer others' preferences based on their choices. In the risky choice task, LLMs using chain-of-thought prompting showed higher correlations with rational models than with human choices. In the inference task, LLMs' inferences aligned with the assumption that people are rational, and these inferences were highly correlated with those made by humans. The study also found that LLMs' assumptions about human rationality align with the human expectation that others act rationally, rather than with how people actually behave. This could lead to misaligned representations of humans, which may affect the development of safe and beneficial AI systems. The research highlights the need for further investigation into how LLMs model human decision-making and the implications for aligning AI with human behavior.Large Language Models (LLMs) often assume that people are more rational than they actually are, according to a study. The research compared LLMs' predictions and simulations of human decisions with a large dataset of human choices. It found that LLMs, including GPT-4o, GPT-4-Turbo, Llama-3-8B, Llama-3-70B, and Claude 3 Opus, tend to align more closely with a rational decision-making model (expected value theory) than with actual human behavior. This suggests that LLMs may not accurately represent how people make decisions, especially in risky choices. The study used two psychological paradigms: a risky choice task where participants choose between gambles, and an inference task where people infer others' preferences based on their choices. In the risky choice task, LLMs using chain-of-thought prompting showed higher correlations with rational models than with human choices. In the inference task, LLMs' inferences aligned with the assumption that people are rational, and these inferences were highly correlated with those made by humans. The study also found that LLMs' assumptions about human rationality align with the human expectation that others act rationally, rather than with how people actually behave. This could lead to misaligned representations of humans, which may affect the development of safe and beneficial AI systems. The research highlights the need for further investigation into how LLMs model human decision-making and the implications for aligning AI with human behavior.