This paper aims to understand the limitations and improve the performance of action recognition algorithms through a systematic evaluation using a thoroughly annotated dataset of human actions. The authors create a dataset called J-HMDB, which includes joint annotations, optical flow, and segmentation for the HMDB51 dataset. They evaluate current methods using this dataset and systematically replace the output of various algorithms with ground truth data to identify the most important factors affecting performance. The results show that high-level pose features, particularly pose over time, significantly outperform low/mid-level features. Additionally, refining low/mid-level features can greatly enhance the accuracy of action recognition frameworks. The paper also highlights the importance of improving optical flow and human detection algorithms. The analysis and J-HMDB dataset are expected to facilitate a deeper understanding of action recognition algorithms and serve as a benchmark for related research.This paper aims to understand the limitations and improve the performance of action recognition algorithms through a systematic evaluation using a thoroughly annotated dataset of human actions. The authors create a dataset called J-HMDB, which includes joint annotations, optical flow, and segmentation for the HMDB51 dataset. They evaluate current methods using this dataset and systematically replace the output of various algorithms with ground truth data to identify the most important factors affecting performance. The results show that high-level pose features, particularly pose over time, significantly outperform low/mid-level features. Additionally, refining low/mid-level features can greatly enhance the accuracy of action recognition frameworks. The paper also highlights the importance of improving optical flow and human detection algorithms. The analysis and J-HMDB dataset are expected to facilitate a deeper understanding of action recognition algorithms and serve as a benchmark for related research.