Structural-RNN is a deep learning approach that combines the power of high-level spatio-temporal graphs with the sequence learning capabilities of Recurrent Neural Networks (RNNs). The method transforms an arbitrary spatio-temporal graph into a rich, scalable, and jointly trainable RNN mixture. This approach is generic and principled, allowing the transformation of any spatio-temporal graph through well-defined steps. The proposed method is evaluated on diverse problems, including human motion modeling, human-object interaction, and driver decision making, showing significant improvements over the state-of-the-art. The method is scalable, fully differentiable, and jointly trainable, enabling the modeling of complex spatio-temporal interactions. The architecture is modular, allowing for the generation of hybrid human motion by combining different motion styles. The S-RNN architecture is shown to outperform unstructured RNNs and structured approaches on various tasks, demonstrating its effectiveness in capturing semantic sub-motions and handling long-term dependencies. The method is validated through extensive experiments, including user studies and visualizations of memory cells, which highlight its ability to learn meaningful semantic interactions. The results show that S-RNN achieves superior performance in tasks such as human activity detection and anticipation, and driver maneuver anticipation, outperforming existing methods in terms of accuracy and generalization. The approach is applicable to a wide range of spatio-temporal problems and provides a flexible and scalable solution for modeling complex interactions.Structural-RNN is a deep learning approach that combines the power of high-level spatio-temporal graphs with the sequence learning capabilities of Recurrent Neural Networks (RNNs). The method transforms an arbitrary spatio-temporal graph into a rich, scalable, and jointly trainable RNN mixture. This approach is generic and principled, allowing the transformation of any spatio-temporal graph through well-defined steps. The proposed method is evaluated on diverse problems, including human motion modeling, human-object interaction, and driver decision making, showing significant improvements over the state-of-the-art. The method is scalable, fully differentiable, and jointly trainable, enabling the modeling of complex spatio-temporal interactions. The architecture is modular, allowing for the generation of hybrid human motion by combining different motion styles. The S-RNN architecture is shown to outperform unstructured RNNs and structured approaches on various tasks, demonstrating its effectiveness in capturing semantic sub-motions and handling long-term dependencies. The method is validated through extensive experiments, including user studies and visualizations of memory cells, which highlight its ability to learn meaningful semantic interactions. The results show that S-RNN achieves superior performance in tasks such as human activity detection and anticipation, and driver maneuver anticipation, outperforming existing methods in terms of accuracy and generalization. The approach is applicable to a wide range of spatio-temporal problems and provides a flexible and scalable solution for modeling complex interactions.