15 Feb 2024 | Olivia Macmillan-Scott, Mirco Musolesi
This paper investigates whether large language models (LLMs) exhibit rational reasoning or irrationality similar to that observed in humans. The study evaluates seven LLMs using cognitive psychology tasks designed to highlight human biases and heuristics. The results show that LLMs display irrationality, but in a manner different from humans. While humans often make errors due to cognitive biases, LLMs frequently provide incorrect answers that do not align with human-like biases. Additionally, LLMs exhibit significant inconsistency in their responses, indicating another form of irrationality. The study also finds that LLMs do not always replicate human biases, as some responses are incorrect due to illogical reasoning rather than cognitive biases. The paper contributes methodologically by proposing a framework to assess and compare the rational reasoning capabilities of LLMs. The results suggest that while some LLMs perform well on certain tasks, others struggle, highlighting the need for further research into the rationality of LLMs. The study also notes that LLMs may not always be able to correctly solve tasks even if they have been exposed to them in training data. The findings have implications for the use of LLMs in critical applications, such as diplomacy and medicine, where consistency and accuracy are essential. The paper concludes that LLMs require further investigation to understand their rationality and to develop benchmarks for evaluating their performance.This paper investigates whether large language models (LLMs) exhibit rational reasoning or irrationality similar to that observed in humans. The study evaluates seven LLMs using cognitive psychology tasks designed to highlight human biases and heuristics. The results show that LLMs display irrationality, but in a manner different from humans. While humans often make errors due to cognitive biases, LLMs frequently provide incorrect answers that do not align with human-like biases. Additionally, LLMs exhibit significant inconsistency in their responses, indicating another form of irrationality. The study also finds that LLMs do not always replicate human biases, as some responses are incorrect due to illogical reasoning rather than cognitive biases. The paper contributes methodologically by proposing a framework to assess and compare the rational reasoning capabilities of LLMs. The results suggest that while some LLMs perform well on certain tasks, others struggle, highlighting the need for further research into the rationality of LLMs. The study also notes that LLMs may not always be able to correctly solve tasks even if they have been exposed to them in training data. The findings have implications for the use of LLMs in critical applications, such as diplomacy and medicine, where consistency and accuracy are essential. The paper concludes that LLMs require further investigation to understand their rationality and to develop benchmarks for evaluating their performance.