4 Jun 2024 | Ehsan Latif, Ramviyas Parasuraman, Xiaoming Zhai
PhysicsAssistant is an interactive learning robot designed to assist students in physics lab investigations using Large Language Models (LLMs) and object detection. The system integrates YOLOv8 for object detection, speech recognition, and a chatbot powered by GPT-3.5-turbo to provide real-time assistance. A user study with ten 8th-grade students evaluated the system's performance against a human expert using Bloom's taxonomy. While GPT-4 outperformed PhysicsAssistant in conceptual and procedural knowledge, PhysicsAssistant demonstrated comparable factual understanding and faster response times. The system's efficiency and cost-effectiveness make it suitable for real-time lab assistance, reducing teacher workload and enhancing student engagement. PhysicsAssistant's multimodal capabilities, including visual and auditory inputs, enable it to provide context-aware responses, improving the learning experience. The system's modular design allows for scalability and adaptability to various educational settings. The integration of LLMs with object detection technology offers a novel approach to educational robotics, enabling interactive and personalized learning. The system's ability to process visual and verbal inputs in real-time, combined with its efficient response time, positions it as a promising tool for K-12 physics education. Future work aims to enhance the system's reasoning capabilities and address complex conceptual questions through improved prompting strategies and domain-specific fine-tuning.PhysicsAssistant is an interactive learning robot designed to assist students in physics lab investigations using Large Language Models (LLMs) and object detection. The system integrates YOLOv8 for object detection, speech recognition, and a chatbot powered by GPT-3.5-turbo to provide real-time assistance. A user study with ten 8th-grade students evaluated the system's performance against a human expert using Bloom's taxonomy. While GPT-4 outperformed PhysicsAssistant in conceptual and procedural knowledge, PhysicsAssistant demonstrated comparable factual understanding and faster response times. The system's efficiency and cost-effectiveness make it suitable for real-time lab assistance, reducing teacher workload and enhancing student engagement. PhysicsAssistant's multimodal capabilities, including visual and auditory inputs, enable it to provide context-aware responses, improving the learning experience. The system's modular design allows for scalability and adaptability to various educational settings. The integration of LLMs with object detection technology offers a novel approach to educational robotics, enabling interactive and personalized learning. The system's ability to process visual and verbal inputs in real-time, combined with its efficient response time, positions it as a promising tool for K-12 physics education. Future work aims to enhance the system's reasoning capabilities and address complex conceptual questions through improved prompting strategies and domain-specific fine-tuning.