Experimenting a New Programming Practice with LLMs

Experimenting a New Programming Practice with LLMs

2017 | Simiao Zhang, Jiaping Wang, Guoliang Dong, Jun Sun, Yueling Zhang, Geguang Pu
This paper introduces AISD, an AI-aided software development framework that engages users throughout the development process, particularly during requirement analysis, high-level system design, and system validation. Unlike existing approaches, AISD involves users in iterative feedback loops, ensuring that the generated system aligns with user expectations. AISD is designed to generate use cases and system designs based on high-level requirements, then iteratively refine the system through user feedback and automated testing. The framework is evaluated using a novel benchmark, CAASD, which includes tasks with non-trivial requirements. The results show that AISD achieves a high pass rate (75.2%) with significantly lower token consumption compared to existing baselines like ChatDev and MetaGPT. The study highlights the importance of user engagement in AI-aided software development, demonstrating that involving users in the development process leads to more accurate and effective systems. AISD's approach emphasizes the need for human oversight in complex tasks, as LLMs may struggle to fully understand and implement requirements without user input. The framework is implemented as a self-contained toolkit, and the results suggest that AISD is both effective and efficient in software development. The study also shows that the number of user interactions required for successful development is relatively low, with an average of four revisions needed for most tasks. Overall, AISD demonstrates the potential of AI-aided software development to reduce the need for manual coding while maintaining the accuracy and effectiveness of the resulting systems.This paper introduces AISD, an AI-aided software development framework that engages users throughout the development process, particularly during requirement analysis, high-level system design, and system validation. Unlike existing approaches, AISD involves users in iterative feedback loops, ensuring that the generated system aligns with user expectations. AISD is designed to generate use cases and system designs based on high-level requirements, then iteratively refine the system through user feedback and automated testing. The framework is evaluated using a novel benchmark, CAASD, which includes tasks with non-trivial requirements. The results show that AISD achieves a high pass rate (75.2%) with significantly lower token consumption compared to existing baselines like ChatDev and MetaGPT. The study highlights the importance of user engagement in AI-aided software development, demonstrating that involving users in the development process leads to more accurate and effective systems. AISD's approach emphasizes the need for human oversight in complex tasks, as LLMs may struggle to fully understand and implement requirements without user input. The framework is implemented as a self-contained toolkit, and the results suggest that AISD is both effective and efficient in software development. The study also shows that the number of user interactions required for successful development is relatively low, with an average of four revisions needed for most tasks. Overall, AISD demonstrates the potential of AI-aided software development to reduce the need for manual coding while maintaining the accuracy and effectiveness of the resulting systems.
Reach us at info@study.space
Understanding Experimenting a New Programming Practice with LLMs