March 20–23, 2024 | Yi Yin Shen, Xinyi Ai, Adalbert Gerald Soosai Raj, Rogers Jeffrey Leo John, Meenakshi Syamkumar
ChatGPT, a conversational AI platform, can generate code to solve problems when given natural language prompts. This study evaluates ChatGPT's performance on assignments from three Data Science (DS) courses with varying difficulty levels. The results show that ChatGPT performs well on assignments with detailed dataset descriptions and progressive question prompts, which divide programming requirements into sub-problems. However, it struggles with abstract prompts that require contextual information. Through prompt engineering techniques such as breaking down questions into steps, providing dataset descriptions, including algorithmic details, and removing extraneous information, ChatGPT's performance significantly improves. These findings suggest that prompt engineering techniques should be integrated into DS curricula to enhance students' problem-solving abilities. The study also highlights the importance of course content consumption for students, even when using AI tools for code generation. Overall, ChatGPT's ability to generate code for DS assignments is influenced by the clarity and specificity of the prompts, and prompt engineering can help students effectively utilize AI tools for data analysis tasks.ChatGPT, a conversational AI platform, can generate code to solve problems when given natural language prompts. This study evaluates ChatGPT's performance on assignments from three Data Science (DS) courses with varying difficulty levels. The results show that ChatGPT performs well on assignments with detailed dataset descriptions and progressive question prompts, which divide programming requirements into sub-problems. However, it struggles with abstract prompts that require contextual information. Through prompt engineering techniques such as breaking down questions into steps, providing dataset descriptions, including algorithmic details, and removing extraneous information, ChatGPT's performance significantly improves. These findings suggest that prompt engineering techniques should be integrated into DS curricula to enhance students' problem-solving abilities. The study also highlights the importance of course content consumption for students, even when using AI tools for code generation. Overall, ChatGPT's ability to generate code for DS assignments is influenced by the clarity and specificity of the prompts, and prompt engineering can help students effectively utilize AI tools for data analysis tasks.