21 February 2024 | Tomaž Kosar, Dragana Ostojić, Yu David Liu, Marjan Mernik
This study explores the impact of using ChatGPT on the learning and performance of novice programmers in a first-year undergraduate object-oriented programming course. The researchers conducted a controlled experiment with 182 participants, dividing them into two groups: one using ChatGPT for practical programming assignments and the other not. The results showed no significant difference in performance, grading results, or midterm exam scores between the two groups (p-values 0.730, 0.760, and 0.856, respectively). The study suggests that ChatGPT can be safely integrated into the education process with specific adjustments, such as designing assignments to minimize direct answers from ChatGPT and implementing an interactive defense process for assignments. The findings indicate that while ChatGPT can enhance learning and engagement, it should be used judiciously to avoid over-reliance and maintain critical thinking skills. The paper discusses the background of large language models, related work, and the experimental design, and provides a detailed analysis of the results and feedback from students.This study explores the impact of using ChatGPT on the learning and performance of novice programmers in a first-year undergraduate object-oriented programming course. The researchers conducted a controlled experiment with 182 participants, dividing them into two groups: one using ChatGPT for practical programming assignments and the other not. The results showed no significant difference in performance, grading results, or midterm exam scores between the two groups (p-values 0.730, 0.760, and 0.856, respectively). The study suggests that ChatGPT can be safely integrated into the education process with specific adjustments, such as designing assignments to minimize direct answers from ChatGPT and implementing an interactive defense process for assignments. The findings indicate that while ChatGPT can enhance learning and engagement, it should be used judiciously to avoid over-reliance and maintain critical thinking skills. The paper discusses the background of large language models, related work, and the experimental design, and provides a detailed analysis of the results and feedback from students.