21 May 2024 | Ranim Khojah, Mazen Mohamad, Philipp Leitner, Francisco Gomes de Oliveira Neto
This paper presents an observational study of 24 professional software engineers who used ChatGPT (GPT-3.5) over a week in their daily work. The study aims to understand how software engineers interact with LLMs like ChatGPT and what factors influence their experience in terms of perceived usefulness and trust. The researchers analyzed the dialogues and exit surveys to identify three main purposes of interaction: artifact manipulation, expert consultation, and training. They found that most participants used ChatGPT for expert consultation, followed by artifact manipulation, and training was the least common. The study also identified internal factors such as prompt structure, personality, and expectations, and external factors like company policies and legal concerns, which influenced the participants' experiences. The results show that participants found ChatGPT helpful for learning new concepts and making better decisions, but less so for team discussions or maintaining focus. Additionally, participants who used ChatGPT for expert consultation found it useful for reducing repetitive tasks, while those using it for training found it helpful for learning. The study concludes with a theoretical framework that categorizes user purposes and identifies factors affecting the personal experience of interacting with ChatGPT, providing insights for future research and practical applications in software engineering.This paper presents an observational study of 24 professional software engineers who used ChatGPT (GPT-3.5) over a week in their daily work. The study aims to understand how software engineers interact with LLMs like ChatGPT and what factors influence their experience in terms of perceived usefulness and trust. The researchers analyzed the dialogues and exit surveys to identify three main purposes of interaction: artifact manipulation, expert consultation, and training. They found that most participants used ChatGPT for expert consultation, followed by artifact manipulation, and training was the least common. The study also identified internal factors such as prompt structure, personality, and expectations, and external factors like company policies and legal concerns, which influenced the participants' experiences. The results show that participants found ChatGPT helpful for learning new concepts and making better decisions, but less so for team discussions or maintaining focus. Additionally, participants who used ChatGPT for expert consultation found it useful for reducing repetitive tasks, while those using it for training found it helpful for learning. The study concludes with a theoretical framework that categorizes user purposes and identifies factors affecting the personal experience of interacting with ChatGPT, providing insights for future research and practical applications in software engineering.