The paper "Generative Students: Using LLM-Simulated Student Profiles to Support Question Item Evaluation" by Xinyi Lu and Xu Wang from the University of Michigan explores the use of large language models (LLMs) to simulate student profiles and generate responses to multiple-choice questions (MCQs). The authors propose a prompt architecture called Generative Students, which is based on the Knowledge-Learning-Instruction (KLI) framework. This architecture defines a generative student profile as a function of the knowledge components (KCs) the student has mastered, is confused about, or has no evidence of knowledge of. The study focuses on heuristic evaluation, a usability inspection method, and uses 45 generative students created with GPT-4 to answer 20 MCQs. The generative students' responses are compared to those of real students, and a high correlation (r=0.72) is found between the generative students' and real students' responses. Additionally, there is significant overlap in the difficult questions identified by both groups. The authors also conduct a case study where an instructor uses the signals provided by the generative students to improve question quality. The results show a significant improvement in the average question score after revising the questions based on the generative students' responses. The paper discusses the potential of using generative students to support rapid prototyping and iteration of questions, highlighting the need for instructor input to steer the process. The study provides insights into creating LLM agents that can simulate specific knowledge deficiencies, making it a promising approach for educational assessment and question generation.The paper "Generative Students: Using LLM-Simulated Student Profiles to Support Question Item Evaluation" by Xinyi Lu and Xu Wang from the University of Michigan explores the use of large language models (LLMs) to simulate student profiles and generate responses to multiple-choice questions (MCQs). The authors propose a prompt architecture called Generative Students, which is based on the Knowledge-Learning-Instruction (KLI) framework. This architecture defines a generative student profile as a function of the knowledge components (KCs) the student has mastered, is confused about, or has no evidence of knowledge of. The study focuses on heuristic evaluation, a usability inspection method, and uses 45 generative students created with GPT-4 to answer 20 MCQs. The generative students' responses are compared to those of real students, and a high correlation (r=0.72) is found between the generative students' and real students' responses. Additionally, there is significant overlap in the difficult questions identified by both groups. The authors also conduct a case study where an instructor uses the signals provided by the generative students to improve question quality. The results show a significant improvement in the average question score after revising the questions based on the generative students' responses. The paper discusses the potential of using generative students to support rapid prototyping and iteration of questions, highlighting the need for instructor input to steer the process. The study provides insights into creating LLM agents that can simulate specific knowledge deficiencies, making it a promising approach for educational assessment and question generation.