Better Call GPT, Comparing Large Language Models Against Lawyers

Better Call GPT, Comparing Large Language Models Against Lawyers

January 2024 | LAUREN MARTIN, NICK WHITEHOUSE, STEPHANIE YIU, LIZZIE CATTERTSON, RIVINDU PERERA
This paper compares Large Language Models (LLMs) with traditional legal contract reviewers, including Junior Lawyers and Legal Process Outsourcers (LPOs), to assess their performance in accuracy, speed, and cost-efficiency during contract review. The study benchmarks LLMs against a ground truth set determined by Senior Lawyers, revealing that advanced models match or exceed human accuracy in identifying legal issues. LLMs complete reviews in seconds, significantly faster than human counterparts, and operate at a fraction of the cost, achieving a 99.97% reduction in cost compared to traditional methods. These results indicate a potential seismic shift in legal practice, with LLMs poised to enhance legal services' accessibility and efficiency. The research explores three key questions: whether LLMs outperform humans in determining and locating legal issues, whether they can review contracts faster, and whether they are cheaper. The findings show that LLMs perform comparably to LPOs and slightly better than Junior Lawyers in issue determination, but LPOs outperform LLMs in issue location. LLMs, however, are significantly faster and cheaper, with the fastest LLM completing reviews in under a minute, while LPOs take over 200 minutes. The cost analysis reveals that LLMs are substantially cheaper, with the fastest LLM costing just 2 cents per contract, compared to $74 for a Junior Lawyer. The study also highlights the importance of selecting the right model for specific tasks and the potential for LLMs to disrupt the legal industry by offering faster, more accurate, and cheaper contract reviews. While LLMs show promise, the research cautions against over-reliance on them, emphasizing the need for ongoing supervision and quality control. The implications for the legal industry are significant, with LLMs likely to reshape legal services, offering greater efficiency and scalability. The research underscores the need for further studies to evaluate LLMs across a broader range of contract types and to explore their capabilities in contract negotiation. Overall, the study supports the assertion that LLMs are viable and superior tools for legal contract review, offering significant advantages in speed, accuracy, and cost.This paper compares Large Language Models (LLMs) with traditional legal contract reviewers, including Junior Lawyers and Legal Process Outsourcers (LPOs), to assess their performance in accuracy, speed, and cost-efficiency during contract review. The study benchmarks LLMs against a ground truth set determined by Senior Lawyers, revealing that advanced models match or exceed human accuracy in identifying legal issues. LLMs complete reviews in seconds, significantly faster than human counterparts, and operate at a fraction of the cost, achieving a 99.97% reduction in cost compared to traditional methods. These results indicate a potential seismic shift in legal practice, with LLMs poised to enhance legal services' accessibility and efficiency. The research explores three key questions: whether LLMs outperform humans in determining and locating legal issues, whether they can review contracts faster, and whether they are cheaper. The findings show that LLMs perform comparably to LPOs and slightly better than Junior Lawyers in issue determination, but LPOs outperform LLMs in issue location. LLMs, however, are significantly faster and cheaper, with the fastest LLM completing reviews in under a minute, while LPOs take over 200 minutes. The cost analysis reveals that LLMs are substantially cheaper, with the fastest LLM costing just 2 cents per contract, compared to $74 for a Junior Lawyer. The study also highlights the importance of selecting the right model for specific tasks and the potential for LLMs to disrupt the legal industry by offering faster, more accurate, and cheaper contract reviews. While LLMs show promise, the research cautions against over-reliance on them, emphasizing the need for ongoing supervision and quality control. The implications for the legal industry are significant, with LLMs likely to reshape legal services, offering greater efficiency and scalability. The research underscores the need for further studies to evaluate LLMs across a broader range of contract types and to explore their capabilities in contract negotiation. Overall, the study supports the assertion that LLMs are viable and superior tools for legal contract review, offering significant advantages in speed, accuracy, and cost.
Reach us at info@study.space
[slides and audio] Better Call GPT%2C Comparing Large Language Models Against Lawyers