TAPTR: Tracking Any Point with Transformers as Detection

TAPTR: Tracking Any Point with Transformers as Detection

19 Mar 2024 | Hongyang Li, Hao Zhang, Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Lei Zhang
The paper introduces TAPTR, a novel framework for Tracking Any Point (TAP) using Transformers. Inspired by DETR-like algorithms, TAPTR represents each tracking point as a query in each video frame, consisting of positional and content features. These queries are updated layer by layer, and their visibility is predicted by the updated content features. Queries from the same tracking point can exchange information through self-attention along the temporal dimension. The framework incorporates cost volume from optical flow models to provide long-term temporal information while mitigating feature drifting. Experiments on various TAP datasets demonstrate superior performance and faster inference speed compared to state-of-the-art methods. The paper also includes extensive ablation studies to validate the effectiveness of each component in the TAPTR framework.The paper introduces TAPTR, a novel framework for Tracking Any Point (TAP) using Transformers. Inspired by DETR-like algorithms, TAPTR represents each tracking point as a query in each video frame, consisting of positional and content features. These queries are updated layer by layer, and their visibility is predicted by the updated content features. Queries from the same tracking point can exchange information through self-attention along the temporal dimension. The framework incorporates cost volume from optical flow models to provide long-term temporal information while mitigating feature drifting. Experiments on various TAP datasets demonstrate superior performance and faster inference speed compared to state-of-the-art methods. The paper also includes extensive ablation studies to validate the effectiveness of each component in the TAPTR framework.
Reach us at info@study.space
[slides and audio] TAPTR%3A Tracking Any Point with Transformers as Detection