Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Camera

Mocap Everyone Everywhere: Lightweight Motion Capture With Smartwatches and a Head-Mounted Camera

6 May 2024 | Jiye Lee, Hanbyul Joo
The paper presents a lightweight and affordable motion capture method that utilizes two smartwatches and a head-mounted camera. This approach is significantly more cost-effective and convenient compared to existing methods that require six or more expert-level IMU devices. The method aims to make wearable motion capture accessible to everyone, enabling 3D full-body motion capture in diverse environments. Key innovations include integrating 6D head poses from the head-mounted camera into the motion estimation pipeline to overcome the sparsity and ambiguities of sensor inputs. An algorithm is proposed to track and update floor levels, enabling capture in expansive indoor and outdoor scenes. A multi-stage Transformer-based regression module is used for motion estimation, and a motion optimization module leverages visual cues from egocentric images to enhance capture quality. The method is evaluated on various challenging scenarios, including complex outdoor environments and everyday motions involving object and social interactions. The results demonstrate superior performance compared to state-of-the-art methods, both in terms of accuracy and robustness.The paper presents a lightweight and affordable motion capture method that utilizes two smartwatches and a head-mounted camera. This approach is significantly more cost-effective and convenient compared to existing methods that require six or more expert-level IMU devices. The method aims to make wearable motion capture accessible to everyone, enabling 3D full-body motion capture in diverse environments. Key innovations include integrating 6D head poses from the head-mounted camera into the motion estimation pipeline to overcome the sparsity and ambiguities of sensor inputs. An algorithm is proposed to track and update floor levels, enabling capture in expansive indoor and outdoor scenes. A multi-stage Transformer-based regression module is used for motion estimation, and a motion optimization module leverages visual cues from egocentric images to enhance capture quality. The method is evaluated on various challenging scenarios, including complex outdoor environments and everyday motions involving object and social interactions. The results demonstrate superior performance compared to state-of-the-art methods, both in terms of accuracy and robustness.
Reach us at info@study.space
[slides and audio] Mocap Everyone Everywhere%3A Lightweight Motion Capture with Smartwatches and a Head-Mounted Camera