July 8–10, 2024, Milan, Italy | Paul Denny, Stephen MacNeil, Jaromir Savelka, Leo Porter, Andrew Luxton-Reilly
The paper explores the characteristics that students value in AI Teaching Assistants (TAs) for programming education. It highlights the challenges of providing timely and personalized feedback to large numbers of students, particularly in programming courses, and the limitations of relying solely on human TAs. The advent of powerful large language models (LLMs) has led to the development of digital TAs that can provide instant, equitable, and round-the-clock support. However, the effectiveness of these tools depends on their ability to promote meaningful learning experiences.
The study deployed an LLM-powered digital assistant, CodeHelp, in an introductory programming course at the University of Auckland. Students provided feedback on the tool's characteristics, focusing on three key themes: scaffolding, appropriateness, and user experience. Scaffolding refers to the need for step-by-step guidance and problem-solving support rather than direct solutions. Appropriateness emphasizes the importance of clear and concise explanations tailored to students' expertise levels. User experience highlights the value of rapid responses, always-available support, and accurate feedback.
The results show that students valued the tool for its ability to provide engaging and correct support, especially during peak times like before assessment deadlines. They preferred features that allowed them to retain autonomy in their learning journey, such as scaffolding that guided them through problem-solving steps. The study also found that students were more likely to seek help when they felt the tool was helpful and accurate, indicating a strong preference for solutions that were not overly constraining.
The paper concludes with design recommendations for AI TAs, emphasizing the importance of preserving students' agency, tailoring responses to their expertise, and providing clear and concise feedback. These findings underscore the potential of AI TAs to enhance learning experiences in programming education, provided they are designed to support rather than replace human TAs.The paper explores the characteristics that students value in AI Teaching Assistants (TAs) for programming education. It highlights the challenges of providing timely and personalized feedback to large numbers of students, particularly in programming courses, and the limitations of relying solely on human TAs. The advent of powerful large language models (LLMs) has led to the development of digital TAs that can provide instant, equitable, and round-the-clock support. However, the effectiveness of these tools depends on their ability to promote meaningful learning experiences.
The study deployed an LLM-powered digital assistant, CodeHelp, in an introductory programming course at the University of Auckland. Students provided feedback on the tool's characteristics, focusing on three key themes: scaffolding, appropriateness, and user experience. Scaffolding refers to the need for step-by-step guidance and problem-solving support rather than direct solutions. Appropriateness emphasizes the importance of clear and concise explanations tailored to students' expertise levels. User experience highlights the value of rapid responses, always-available support, and accurate feedback.
The results show that students valued the tool for its ability to provide engaging and correct support, especially during peak times like before assessment deadlines. They preferred features that allowed them to retain autonomy in their learning journey, such as scaffolding that guided them through problem-solving steps. The study also found that students were more likely to seek help when they felt the tool was helpful and accurate, indicating a strong preference for solutions that were not overly constraining.
The paper concludes with design recommendations for AI TAs, emphasizing the importance of preserving students' agency, tailoring responses to their expertise, and providing clear and concise feedback. These findings underscore the potential of AI TAs to enhance learning experiences in programming education, provided they are designed to support rather than replace human TAs.