Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models

Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models

April 25, 2024 | Matthew Dahl, Varun Magesh, Mirac Suzgun, Daniel E. Ho
The article "Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models" by Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E. Ho explores the extent and nature of "hallucinations" in large language models (LLMs) when dealing with legal information. The authors present the first systematic evidence of these hallucinations, documenting trends across jurisdictions, courts, time periods, and cases. Using OpenAI’s ChatGPT 4 and other public models, they find that LLMs hallucinate at least 58% of the time, struggle to predict their own hallucinations, and often uncritically accept users’ incorrect legal assumptions. The study highlights the risks of rapid and unsupervised integration of LLMs into legal tasks and develops a typology of legal hallucinations to guide future research. The findings suggest that while LLMs can offer accessibility and affordability in legal information and services, their current shortcomings, particularly in generating accurate and reliable statements of the law, significantly hinder their effectiveness. The article contributes to the literature on the intersection of law and technology, the implications of AI for access to justice, and the pressing issue of algorithmic harm.The article "Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models" by Matthew Dahl, Varun Magesh, Mirac Suzgun, and Daniel E. Ho explores the extent and nature of "hallucinations" in large language models (LLMs) when dealing with legal information. The authors present the first systematic evidence of these hallucinations, documenting trends across jurisdictions, courts, time periods, and cases. Using OpenAI’s ChatGPT 4 and other public models, they find that LLMs hallucinate at least 58% of the time, struggle to predict their own hallucinations, and often uncritically accept users’ incorrect legal assumptions. The study highlights the risks of rapid and unsupervised integration of LLMs into legal tasks and develops a typology of legal hallucinations to guide future research. The findings suggest that while LLMs can offer accessibility and affordability in legal information and services, their current shortcomings, particularly in generating accurate and reliable statements of the law, significantly hinder their effectiveness. The article contributes to the literature on the intersection of law and technology, the implications of AI for access to justice, and the pressing issue of algorithmic harm.
Reach us at info@study.space
[slides and audio] Large Legal Fictions%3A Profiling Legal Hallucinations in Large Language Models