This paper explores the integration of linear state-space models (SSMs) into deep neural network architectures, particularly in the context of foundation models. SSMs, which are widely used in control theory to model dynamical systems, offer a promising alternative to the Transformer architecture, especially for long-sequence tasks. The paper provides a control-theoretic overview of SSM-based architectures, highlighting their potential to overcome the limitations of Transformers, such as computational inefficiency and poor scalability for long contexts. SSMs are characterized by their recurrent nature, allowing them to capture information from past inputs efficiently. They are also computationally efficient to train and infer, unlike traditional recurrent neural networks (RNNs). Recent models like Mamba have demonstrated superior performance on long-context tasks, such as the Long Range Arena (LRA) benchmark, compared to state-of-the-art Transformers. The paper reviews various SSM proposals, including S4, S4D, S5, LRU, S6, and RG-LRU, discussing their parametrization, discretization, structure, initialization, and implementation. These models differ in their handling of time-varying dynamics and their ability to capture long-range memory, which is crucial for effective sequence modeling. The study evaluates the performance of these SSMs on the LRA benchmark, finding that LTI-based models (S4, S4D, S5, LRU) outperform LTV-based models (S6, RG-LRU) and Transformers. This result is surprising from a control-theoretic perspective, as LTV systems should theoretically perform at least as well as LTI systems. However, the specific time-varying parametrization of S6 and RG-LRU leads to suboptimal performance. The paper concludes that SSMs, particularly LTV versions, have significant potential in foundation models and systems and control theory. They offer opportunities for improved explainability, design, and performance by leveraging existing system-theoretic results. Future research should focus on understanding the role of eigenvalues in LTV-based models and developing more effective parametrizations to achieve performance comparable to LTI models.This paper explores the integration of linear state-space models (SSMs) into deep neural network architectures, particularly in the context of foundation models. SSMs, which are widely used in control theory to model dynamical systems, offer a promising alternative to the Transformer architecture, especially for long-sequence tasks. The paper provides a control-theoretic overview of SSM-based architectures, highlighting their potential to overcome the limitations of Transformers, such as computational inefficiency and poor scalability for long contexts. SSMs are characterized by their recurrent nature, allowing them to capture information from past inputs efficiently. They are also computationally efficient to train and infer, unlike traditional recurrent neural networks (RNNs). Recent models like Mamba have demonstrated superior performance on long-context tasks, such as the Long Range Arena (LRA) benchmark, compared to state-of-the-art Transformers. The paper reviews various SSM proposals, including S4, S4D, S5, LRU, S6, and RG-LRU, discussing their parametrization, discretization, structure, initialization, and implementation. These models differ in their handling of time-varying dynamics and their ability to capture long-range memory, which is crucial for effective sequence modeling. The study evaluates the performance of these SSMs on the LRA benchmark, finding that LTI-based models (S4, S4D, S5, LRU) outperform LTV-based models (S6, RG-LRU) and Transformers. This result is surprising from a control-theoretic perspective, as LTV systems should theoretically perform at least as well as LTI systems. However, the specific time-varying parametrization of S6 and RG-LRU leads to suboptimal performance. The paper concludes that SSMs, particularly LTV versions, have significant potential in foundation models and systems and control theory. They offer opportunities for improved explainability, design, and performance by leveraging existing system-theoretic results. Future research should focus on understanding the role of eigenvalues in LTV-based models and developing more effective parametrizations to achieve performance comparable to LTI models.