This paper explores the integration of generative AI, specifically GPT-4, into an automated assessment tool (AAT) for enhancing programming error messages in a CS1 course at Abilene Christian University. The study aims to address the common issue of cryptic and difficult-to-understand error messages by providing real-time feedback from ChatGPT for compiler, run-time, and logic errors. The authors implemented this integration during the Fall 2023 semester and collected submission data and survey responses to evaluate the effectiveness and user perception of the AI feedback. Key findings include: 1. **Submission Behavior**: Despite the hypothesis that real-time AI feedback would reduce the number of submissions, the mean number of submissions in 2023 was higher than in previous years, suggesting that the AI feedback did not significantly impact submission rates. 2. **User Perception**: Students had mixed reactions to the AI feedback. While some found it helpful, others found it vague or incorrect. Most students were interested in using AI feedback again, indicating a potential need for more nuanced and context-specific feedback. 3. **Design Considerations**: The interaction style of the AI feedback was crucial. Students desired a more conversational interface where they could ask follow-up questions and receive more detailed and specific advice. The study concludes that while generative AI can enhance error messages, the design of the interface and the interaction style are critical factors in determining the usability and effectiveness of such tools. Future research should focus on designing more interactive and context-aware interfaces to improve the user experience.This paper explores the integration of generative AI, specifically GPT-4, into an automated assessment tool (AAT) for enhancing programming error messages in a CS1 course at Abilene Christian University. The study aims to address the common issue of cryptic and difficult-to-understand error messages by providing real-time feedback from ChatGPT for compiler, run-time, and logic errors. The authors implemented this integration during the Fall 2023 semester and collected submission data and survey responses to evaluate the effectiveness and user perception of the AI feedback. Key findings include: 1. **Submission Behavior**: Despite the hypothesis that real-time AI feedback would reduce the number of submissions, the mean number of submissions in 2023 was higher than in previous years, suggesting that the AI feedback did not significantly impact submission rates. 2. **User Perception**: Students had mixed reactions to the AI feedback. While some found it helpful, others found it vague or incorrect. Most students were interested in using AI feedback again, indicating a potential need for more nuanced and context-specific feedback. 3. **Design Considerations**: The interaction style of the AI feedback was crucial. Students desired a more conversational interface where they could ask follow-up questions and receive more detailed and specific advice. The study concludes that while generative AI can enhance error messages, the design of the interface and the interaction style are critical factors in determining the usability and effectiveness of such tools. Future research should focus on designing more interactive and context-aware interfaces to improve the user experience.