Demystifying LLM-based Software Engineering Agents

Demystifying LLM-based Software Engineering Agents

1 Jul 2024 | Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, Lingming Zhang
This paper introduces AGENTLESS, an agentless approach to automatically solve software development problems. AGENTLESS uses a two-phase process of localization followed by repair, without allowing the LLM to decide future actions or operate with complex tools. The approach is evaluated on the SWE-bench Lite benchmark, where AGENTLESS achieves the highest performance (27.33%) and the lowest cost ($0.34) compared to existing open-source software agents. The results show that AGENTLESS can effectively solve software development problems with a simple and interpretable approach. The paper also highlights the current overlooked potential of such techniques in autonomous software development. AGENTLESS is open-sourced at https://github.com/OpenAutoCoder/Agentless. The paper also discusses the limitations of agent-based approaches, including complex tool usage, lack of control in decision planning, and limited ability to self-reflect. The paper proposes a simple hierarchical localization process to identify the correct code locations and a simple diff format for generating patches. The paper also discusses the evaluation of AGENTLESS on the SWE-bench Lite benchmark and the construction of a more rigorous benchmark, SWE-bench Lite-S, by excluding problematic issues. The paper concludes that AGENTLESS offers a simple and cost-effective solution to tackle real-world software engineering issues and highlights the potential of a simple, interpretable technique in autonomous software development.This paper introduces AGENTLESS, an agentless approach to automatically solve software development problems. AGENTLESS uses a two-phase process of localization followed by repair, without allowing the LLM to decide future actions or operate with complex tools. The approach is evaluated on the SWE-bench Lite benchmark, where AGENTLESS achieves the highest performance (27.33%) and the lowest cost ($0.34) compared to existing open-source software agents. The results show that AGENTLESS can effectively solve software development problems with a simple and interpretable approach. The paper also highlights the current overlooked potential of such techniques in autonomous software development. AGENTLESS is open-sourced at https://github.com/OpenAutoCoder/Agentless. The paper also discusses the limitations of agent-based approaches, including complex tool usage, lack of control in decision planning, and limited ability to self-reflect. The paper proposes a simple hierarchical localization process to identify the correct code locations and a simple diff format for generating patches. The paper also discusses the evaluation of AGENTLESS on the SWE-bench Lite benchmark and the construction of a more rigorous benchmark, SWE-bench Lite-S, by excluding problematic issues. The paper concludes that AGENTLESS offers a simple and cost-effective solution to tackle real-world software engineering issues and highlights the potential of a simple, interpretable technique in autonomous software development.
Reach us at info@study.space