A Performance Study of LLM-Generated Code on Leetcode

A Performance Study of LLM-Generated Code on Leetcode

June 18–21, 2024, Salerno, Italy | Tristan Coignion, Clément Quinton, Romain Rouvoy
This study evaluates the efficiency of code generation by Large Language Models (LLMs) using a dataset from LeetCode, comparing 18 LLMs and considering factors such as model temperature and success rate. The research introduces a novel method for measuring and comparing the speed of LLM-generated code, revealing that LLMs produce code with comparable performance, regardless of the adopted LLM. The study also finds that LLMs can generate code that is, on average, more efficient than human-written code. The paper discusses the use of LeetCode as a benchmarking dataset, the limitations of potential data contamination, and the reliability of the platform's measurements. The findings contribute to a better understanding of LLM capabilities in code generation and set the stage for future optimizations in the field. The study concludes by highlighting the importance of avoiding data contamination and using newer LeetCode problems to ensure the validity of LLM evaluations.This study evaluates the efficiency of code generation by Large Language Models (LLMs) using a dataset from LeetCode, comparing 18 LLMs and considering factors such as model temperature and success rate. The research introduces a novel method for measuring and comparing the speed of LLM-generated code, revealing that LLMs produce code with comparable performance, regardless of the adopted LLM. The study also finds that LLMs can generate code that is, on average, more efficient than human-written code. The paper discusses the use of LeetCode as a benchmarking dataset, the limitations of potential data contamination, and the reliability of the platform's measurements. The findings contribute to a better understanding of LLM capabilities in code generation and set the stage for future optimizations in the field. The study concludes by highlighting the importance of avoiding data contamination and using newer LeetCode problems to ensure the validity of LLM evaluations.
Reach us at info@study.space