The paper presents the Structured Segment Network (SSN), a novel framework for temporal action detection in untrimmed videos. SSN models the temporal structure of each action instance using a structured temporal pyramid, which includes three stages: starting, course, and ending. On top of this pyramid, a decomposed discriminative model with two classifiers is introduced: one for classifying actions and another for determining completeness. This allows the framework to distinguish between positive proposals and background or incomplete ones, leading to accurate recognition and localization. The components are integrated into a unified network that can be trained end-to-end efficiently. Additionally, a temporal action proposal scheme called Temporal Actionness Grouping (TAG) is proposed to generate high-quality action proposals. The method outperforms previous state-of-the-art methods on the THUMOS14 and ActivityNet datasets, demonstrating superior accuracy and adaptability in handling various temporal structures of actions.The paper presents the Structured Segment Network (SSN), a novel framework for temporal action detection in untrimmed videos. SSN models the temporal structure of each action instance using a structured temporal pyramid, which includes three stages: starting, course, and ending. On top of this pyramid, a decomposed discriminative model with two classifiers is introduced: one for classifying actions and another for determining completeness. This allows the framework to distinguish between positive proposals and background or incomplete ones, leading to accurate recognition and localization. The components are integrated into a unified network that can be trained end-to-end efficiently. Additionally, a temporal action proposal scheme called Temporal Actionness Grouping (TAG) is proposed to generate high-quality action proposals. The method outperforms previous state-of-the-art methods on the THUMOS14 and ActivityNet datasets, demonstrating superior accuracy and adaptability in handling various temporal structures of actions.