February 11-14, 2024 | Cise Midoglu, Saeed Shafiee Sabet, Mehdi Houshmand Sarkhoosh, Mohammad Majidi, Sushant Gautam, Håkon Maric Solberg, Tomas Kupka, Pål Halvorsen
This paper presents an AI-based multimedia production framework for automatically generating and sharing sports highlights on social media. The framework includes eight key functions: event detection and classification, player detection and tracking, highlight clipping, cropping (aspect ratio retargeting), thumbnail generation, game summarization, caption generation, and social media sharing. The system aims to automate the creation of sports highlights, reducing the manual effort required for content generation and enabling efficient distribution on social media platforms.
Event detection involves identifying and classifying soccer and ice hockey events in real-time using 3D convolutional neural networks. The system achieves high accuracy with low latency, though industry feedback indicates that current performance may not meet the requirements for official events. The integration of audio features can improve detection for certain events, but may be detrimental for others.
Player detection and tracking involve identifying and following players across video sequences, using object detection models and tracking algorithms. The system uses jersey colors for team mapping and optical character recognition to extract player numbers. The pipeline enables persistent tracking and accurate identification of players.
Highlight clipping involves automatically generating clips of specific events, such as goals and bookings, using scene boundary detection, logo detection, and cheering removal. The system outperforms static clipping in user studies, with automated clipping preferred for its quality and variety.
Cropping (aspect ratio retargeting) adjusts video frames to fit different social media aspect ratios, using Points of Interest (POI) such as the soccer ball or hockey puck. The system ensures consistent cropping and maintains visual quality, with user studies showing that SmartCrop consistently yields superior results.
Thumbnail generation selects and ranks video frames as potential thumbnails based on relevance and image quality. The system uses automated selection, which is preferred over static and manual methods, with user feedback highlighting the importance of high image quality, player faces, and action content.
Game summarization and caption generation use audio analysis, structured metadata, and captions to create concise summaries and captions for social media. The system leverages large language models to generate contextually rich and platform-specific outputs.
Social media sharing involves integrating the entire pipeline into a single system for seamless content distribution. The system supports direct sharing to platforms like TikTok and Instagram, reducing the need for manual processing and improving efficiency.
The framework demonstrates the potential of AI in automating sports highlight generation, improving engagement, and enhancing the quality of user experience on social media. Future work includes expanding the system to other sports and improving adaptability to evolving social media APIs.This paper presents an AI-based multimedia production framework for automatically generating and sharing sports highlights on social media. The framework includes eight key functions: event detection and classification, player detection and tracking, highlight clipping, cropping (aspect ratio retargeting), thumbnail generation, game summarization, caption generation, and social media sharing. The system aims to automate the creation of sports highlights, reducing the manual effort required for content generation and enabling efficient distribution on social media platforms.
Event detection involves identifying and classifying soccer and ice hockey events in real-time using 3D convolutional neural networks. The system achieves high accuracy with low latency, though industry feedback indicates that current performance may not meet the requirements for official events. The integration of audio features can improve detection for certain events, but may be detrimental for others.
Player detection and tracking involve identifying and following players across video sequences, using object detection models and tracking algorithms. The system uses jersey colors for team mapping and optical character recognition to extract player numbers. The pipeline enables persistent tracking and accurate identification of players.
Highlight clipping involves automatically generating clips of specific events, such as goals and bookings, using scene boundary detection, logo detection, and cheering removal. The system outperforms static clipping in user studies, with automated clipping preferred for its quality and variety.
Cropping (aspect ratio retargeting) adjusts video frames to fit different social media aspect ratios, using Points of Interest (POI) such as the soccer ball or hockey puck. The system ensures consistent cropping and maintains visual quality, with user studies showing that SmartCrop consistently yields superior results.
Thumbnail generation selects and ranks video frames as potential thumbnails based on relevance and image quality. The system uses automated selection, which is preferred over static and manual methods, with user feedback highlighting the importance of high image quality, player faces, and action content.
Game summarization and caption generation use audio analysis, structured metadata, and captions to create concise summaries and captions for social media. The system leverages large language models to generate contextually rich and platform-specific outputs.
Social media sharing involves integrating the entire pipeline into a single system for seamless content distribution. The system supports direct sharing to platforms like TikTok and Instagram, reducing the need for manual processing and improving efficiency.
The framework demonstrates the potential of AI in automating sports highlight generation, improving engagement, and enhancing the quality of user experience on social media. Future work includes expanding the system to other sports and improving adaptability to evolving social media APIs.