The paper introduces the Structured State Space (S4) model, a novel approach to sequence modeling that efficiently handles long-range dependencies (LRDs) in various modalities and tasks. S4 is based on a structured state space model (SSM) parameterization, which allows for more efficient computation compared to previous methods. The key contributions of S4 include: 1. **Efficient Computation**: S4 uses a low-rank correction to diagonalize the state matrix \(A\), reducing the computational complexity to \(\tilde{O}(N + L)\) for both training and inference, where \(N\) is the state size and \(L\) is the sequence length. 2. **Theoretical and Empirical Performance**: S4 achieves strong empirical results across a diverse range of benchmarks, including: - 91% accuracy on sequential CIFAR-10 with no data augmentation or auxiliary losses. - Significantly closing the gap to Transformers on image and language modeling tasks while being 60× faster. - Achieving state-of-the-art (SoTA) results on the Long Range Arena (LRA) benchmark, including solving the challenging Path-X task of length 16k. 3. **Generalizability**: S4 demonstrates its potential as a general sequence modeling solution by performing well on tasks such as large-scale generative modeling, fast autoregressive generation, sampling resolution change, and learning with weaker inductive biases. 4. **Ablation Studies**: The paper includes ablation studies to validate the importance of the HiPPO initialization and the NPLR parameterization in S4. Overall, S4 offers a principled and efficient approach to sequence modeling, addressing the challenges of long-range dependencies and providing strong performance across multiple benchmarks.The paper introduces the Structured State Space (S4) model, a novel approach to sequence modeling that efficiently handles long-range dependencies (LRDs) in various modalities and tasks. S4 is based on a structured state space model (SSM) parameterization, which allows for more efficient computation compared to previous methods. The key contributions of S4 include: 1. **Efficient Computation**: S4 uses a low-rank correction to diagonalize the state matrix \(A\), reducing the computational complexity to \(\tilde{O}(N + L)\) for both training and inference, where \(N\) is the state size and \(L\) is the sequence length. 2. **Theoretical and Empirical Performance**: S4 achieves strong empirical results across a diverse range of benchmarks, including: - 91% accuracy on sequential CIFAR-10 with no data augmentation or auxiliary losses. - Significantly closing the gap to Transformers on image and language modeling tasks while being 60× faster. - Achieving state-of-the-art (SoTA) results on the Long Range Arena (LRA) benchmark, including solving the challenging Path-X task of length 16k. 3. **Generalizability**: S4 demonstrates its potential as a general sequence modeling solution by performing well on tasks such as large-scale generative modeling, fast autoregressive generation, sampling resolution change, and learning with weaker inductive biases. 4. **Ablation Studies**: The paper includes ablation studies to validate the importance of the HiPPO initialization and the NPLR parameterization in S4. Overall, S4 offers a principled and efficient approach to sequence modeling, addressing the challenges of long-range dependencies and providing strong performance across multiple benchmarks.