July 15-19, 2024 | Dorin Pomian, Abhira Bellur*, Malinda Dilhara, Zarina Kurbatova, Egor Bogomolov, Andrey Sokolov, Timofey Bryksin, Danny Dig
EM-Assist is an IntelliJ IDEA plugin that uses large language models (LLMs) to generate and validate refactoring suggestions for extracting methods. It addresses the challenge of identifying which code fragments to extract, a task that traditional tools often fail to perform effectively. EM-Assist first generates refactoring suggestions using LLMs, then filters out invalid or non-useful suggestions, enhances valid ones using program slicing, and ranks them for presentation to developers. It then applies the selected suggestion within the IDE, ensuring safe execution.
In a comprehensive evaluation of 1,752 real-world refactorings from open-source projects, EM-Assist achieved a recall rate of 53.4% among its top-5 recommendations, outperforming the previous best-in-class tool (JExtract) with a recall rate of 39.4%. A usability survey with 18 industrial developers found that 94.4% rated the tool positively.
EM-Assist supports Java and Kotlin and is extensible for other languages. It leverages the IntelliJ IDE's APIs for validation and refactoring. The tool uses LLMs for developer-aligned suggestions but ensures refactoring safety through the IDE. Limitations include the need to send code to OpenAI, which may affect response time, and restrictions due to company policies against Generative AI.
EM-Assist demonstrates the potential of LLMs as effective refactoring assistants, offering a novel approach to validate and enhance LLM outputs. It bridges the gap between refactoring suggestions and developer practices, providing high-quality, safe refactorings. The tool is designed to be easily adaptable to different LLMs and is a significant advancement in automated refactoring.EM-Assist is an IntelliJ IDEA plugin that uses large language models (LLMs) to generate and validate refactoring suggestions for extracting methods. It addresses the challenge of identifying which code fragments to extract, a task that traditional tools often fail to perform effectively. EM-Assist first generates refactoring suggestions using LLMs, then filters out invalid or non-useful suggestions, enhances valid ones using program slicing, and ranks them for presentation to developers. It then applies the selected suggestion within the IDE, ensuring safe execution.
In a comprehensive evaluation of 1,752 real-world refactorings from open-source projects, EM-Assist achieved a recall rate of 53.4% among its top-5 recommendations, outperforming the previous best-in-class tool (JExtract) with a recall rate of 39.4%. A usability survey with 18 industrial developers found that 94.4% rated the tool positively.
EM-Assist supports Java and Kotlin and is extensible for other languages. It leverages the IntelliJ IDE's APIs for validation and refactoring. The tool uses LLMs for developer-aligned suggestions but ensures refactoring safety through the IDE. Limitations include the need to send code to OpenAI, which may affect response time, and restrictions due to company policies against Generative AI.
EM-Assist demonstrates the potential of LLMs as effective refactoring assistants, offering a novel approach to validate and enhance LLM outputs. It bridges the gap between refactoring suggestions and developer practices, providing high-quality, safe refactorings. The tool is designed to be easily adaptable to different LLMs and is a significant advancement in automated refactoring.