This paper presents a detailed analysis of action recognition algorithms using a new dataset called J-HMDB, which is derived from the HMDB51 dataset. J-HMDB includes detailed annotations of human joints, enabling the derivation of ground truth optical flow and segmentation. The authors evaluate current methods on this dataset and systematically replace the output of various algorithms with ground truth to determine what is important for action recognition. They find that high-level pose features significantly outperform low/mid-level features, particularly pose over time. While current pose estimation algorithms are not perfect, features extracted from estimated pose on a subset of J-HMDB, where the full body is visible, outperform low/mid-level features. The authors also find that refining low/mid-level features can greatly improve the accuracy of the action recognition framework, suggesting the importance of improving optical flow and human detection algorithms. The analysis and J-HMDB dataset are expected to facilitate a deeper understanding of action recognition algorithms. The paper also discusses the impact of different levels of features on action recognition, showing that high-level pose features are the best for action recognition. The study highlights the importance of pose estimation and the need for more accurate human detection algorithms. The results suggest that using ground truth pose information can significantly improve action recognition performance beyond current state-of-the-art methods. The paper also evaluates the impact of different feature types on action recognition, showing that high-level pose features are the most effective. The study provides insights into the importance of pose estimation and the need for more accurate human detection algorithms. The results suggest that using ground truth pose information can significantly improve action recognition performance beyond current state-of-the-art methods.This paper presents a detailed analysis of action recognition algorithms using a new dataset called J-HMDB, which is derived from the HMDB51 dataset. J-HMDB includes detailed annotations of human joints, enabling the derivation of ground truth optical flow and segmentation. The authors evaluate current methods on this dataset and systematically replace the output of various algorithms with ground truth to determine what is important for action recognition. They find that high-level pose features significantly outperform low/mid-level features, particularly pose over time. While current pose estimation algorithms are not perfect, features extracted from estimated pose on a subset of J-HMDB, where the full body is visible, outperform low/mid-level features. The authors also find that refining low/mid-level features can greatly improve the accuracy of the action recognition framework, suggesting the importance of improving optical flow and human detection algorithms. The analysis and J-HMDB dataset are expected to facilitate a deeper understanding of action recognition algorithms. The paper also discusses the impact of different levels of features on action recognition, showing that high-level pose features are the best for action recognition. The study highlights the importance of pose estimation and the need for more accurate human detection algorithms. The results suggest that using ground truth pose information can significantly improve action recognition performance beyond current state-of-the-art methods. The paper also evaluates the impact of different feature types on action recognition, showing that high-level pose features are the most effective. The study provides insights into the importance of pose estimation and the need for more accurate human detection algorithms. The results suggest that using ground truth pose information can significantly improve action recognition performance beyond current state-of-the-art methods.