1 Jul 2024 | Chunqiu Steven Xia*, Yinlin Deng*, Soren Dunn Lingming Zhang
The paper "AGENTLESS 🤖: Demystifying LLM-based Software Engineering Agents" by Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang from the University of Illinois Urbana-Champaign explores the potential of agentless approaches in software development tasks. The authors question whether complex autonomous software agents are necessary and propose AGENTLESS, an agentless approach to solve software development problems. AGENTLESS employs a simple two-phase process: localization followed by repair, without allowing the LLM to make autonomous decisions or use complex tools. The evaluation on the SWE-bench Lite benchmark shows that AGENTLESS achieves the highest performance (27.33%) and lowest cost (60.34%) compared to existing open-source software agents. The authors also conduct a manual analysis of the SWE-bench Lite dataset, identifying issues with ground truth patches, missing information, and misleading solutions. They construct a more rigorous benchmark, SWE-bench Lite-S, by excluding problematic issues. The paper highlights the overlooked potential of simple, interpretable techniques in autonomous software development and aims to reset the baseline for future work. AGENTLESS is open-sourced at <https://github.com/OpenAutoCoder/Agentless>.The paper "AGENTLESS 🤖: Demystifying LLM-based Software Engineering Agents" by Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang from the University of Illinois Urbana-Champaign explores the potential of agentless approaches in software development tasks. The authors question whether complex autonomous software agents are necessary and propose AGENTLESS, an agentless approach to solve software development problems. AGENTLESS employs a simple two-phase process: localization followed by repair, without allowing the LLM to make autonomous decisions or use complex tools. The evaluation on the SWE-bench Lite benchmark shows that AGENTLESS achieves the highest performance (27.33%) and lowest cost (60.34%) compared to existing open-source software agents. The authors also conduct a manual analysis of the SWE-bench Lite dataset, identifying issues with ground truth patches, missing information, and misleading solutions. They construct a more rigorous benchmark, SWE-bench Lite-S, by excluding problematic issues. The paper highlights the overlooked potential of simple, interpretable techniques in autonomous software development and aims to reset the baseline for future work. AGENTLESS is open-sourced at <https://github.com/OpenAutoCoder/Agentless>.