Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice

Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice

21 May 2024 | Ranim Khojah, Mazen Mohamad, Philipp Leitner, Francisco Gomes de Oliveira Neto
This paper presents an observational study of 24 professional software engineers who used ChatGPT over a week in their jobs. The study qualitatively analyzed their dialogues with the chatbot and their experiences through an exit survey. The findings show that software engineers primarily use ChatGPT for guidance on solving tasks or learning about topics, rather than generating ready-to-use software artifacts. The study proposes a theoretical framework that considers the purpose of the interaction, internal factors (e.g., user personality), and external factors (e.g., company policy) to understand how they shape the user's experience in terms of perceived usefulness and trust. The framework is intended to guide future research on LLM usage in software engineering and serve as a reference for designing empirical studies in this area. The study also identifies three main purposes for ChatGPT usage: artifact manipulation (generating or modifying code), expert consultation (seeking guidance), and training (learning). The study further reveals that ChatGPT is most commonly used during the implementation phase of software development, but also in planning, design, and testing. Internal factors such as prompt phrasing and user personality, as well as external factors like company policies, significantly influence the user's experience. The study also highlights that participants often lack sufficient context when using ChatGPT, leading to frustration. The findings suggest that ChatGPT is a useful tool for learning and reducing repetitive tasks, but its effectiveness depends on the user's expectations and the context of use. The study concludes that ChatGPT can be a valuable tool for software engineers, but its use is influenced by various factors, including legal and ethical concerns, company policies, and the user's personal experience.This paper presents an observational study of 24 professional software engineers who used ChatGPT over a week in their jobs. The study qualitatively analyzed their dialogues with the chatbot and their experiences through an exit survey. The findings show that software engineers primarily use ChatGPT for guidance on solving tasks or learning about topics, rather than generating ready-to-use software artifacts. The study proposes a theoretical framework that considers the purpose of the interaction, internal factors (e.g., user personality), and external factors (e.g., company policy) to understand how they shape the user's experience in terms of perceived usefulness and trust. The framework is intended to guide future research on LLM usage in software engineering and serve as a reference for designing empirical studies in this area. The study also identifies three main purposes for ChatGPT usage: artifact manipulation (generating or modifying code), expert consultation (seeking guidance), and training (learning). The study further reveals that ChatGPT is most commonly used during the implementation phase of software development, but also in planning, design, and testing. Internal factors such as prompt phrasing and user personality, as well as external factors like company policies, significantly influence the user's experience. The study also highlights that participants often lack sufficient context when using ChatGPT, leading to frustration. The findings suggest that ChatGPT is a useful tool for learning and reducing repetitive tasks, but its effectiveness depends on the user's expectations and the context of use. The study concludes that ChatGPT can be a valuable tool for software engineers, but its use is influenced by various factors, including legal and ethical concerns, company policies, and the user's personal experience.
Reach us at info@study.space
Understanding Beyond Code Generation%3A An Observational Study of ChatGPT Usage in Software Engineering Practice