March 18-21, 2024 | Peter Andrews, Oda Nordberg, Stephanie Zubicueta Portales, Njål Borch, Frode Guribye, Kazuyuki Fujita, Morten Fjeld
AiCommentator is a Multimodal Conversational Agent (MCA) designed for embedded visualization in football viewing, enhancing user comprehension of player performance and in-game events through automated and interactive commentary. The system integrates embedded visualizations with either non-interactive or interactive commentary modes, leveraging multimodal techniques including computer vision and large language models. The event system infers game states using multi-object tracking and computer vision, enabling automated responsive commentary. The system was evaluated through a mixed-method study with 16 participants, revealing that users preferred the interactive mode for higher engagement and satisfaction. The study addressed three research questions: which mode offers higher engagement, how the modes support young adult viewers' understanding of players and events, and how users perceive the interactive mode's usability. The results showed that the interactive mode was more engaging and effective in enhancing viewer understanding and immersion. The system provides real-time and historical in-game statistics, player locations, and automated commentary, offering a tailored, interactive sports-viewing experience. The paper presents the system's design, user study results, and implications for future research in interactive media for sports broadcasting.AiCommentator is a Multimodal Conversational Agent (MCA) designed for embedded visualization in football viewing, enhancing user comprehension of player performance and in-game events through automated and interactive commentary. The system integrates embedded visualizations with either non-interactive or interactive commentary modes, leveraging multimodal techniques including computer vision and large language models. The event system infers game states using multi-object tracking and computer vision, enabling automated responsive commentary. The system was evaluated through a mixed-method study with 16 participants, revealing that users preferred the interactive mode for higher engagement and satisfaction. The study addressed three research questions: which mode offers higher engagement, how the modes support young adult viewers' understanding of players and events, and how users perceive the interactive mode's usability. The results showed that the interactive mode was more engaging and effective in enhancing viewer understanding and immersion. The system provides real-time and historical in-game statistics, player locations, and automated commentary, offering a tailored, interactive sports-viewing experience. The paper presents the system's design, user study results, and implications for future research in interactive media for sports broadcasting.