This paper explores the integration of linear state-space models (SSMs) into deep neural network architectures, particularly in the context of foundation models like GPT-4. SSMs, which are traditionally used in control theory to model dynamical systems, offer a recurrent nature that captures past inputs efficiently, making them suitable for long-context tasks. The authors provide a gentle introduction to SSM-based architectures from a control theoretical perspective, reviewing the essential components and recent advancements. They highlight the mathematical structure, computational considerations, and initialization strategies of SSMs, emphasizing the importance of memory and eigenvalues in achieving long-range memory. The paper also presents a comparative analysis of various SSM proposals, including S4, S4D, S5, LRU, S6, and RG-LRU, on the Long Range Arena (LRA) benchmark, which evaluates models' reasoning ability and handling of diverse data types. The results show that LTI-based SSMs outperform LTV-based models, raising questions about the theoretical properties and eigenvalue significance in LTV models. The authors conclude by discussing future research opportunities, particularly in understanding the theoretical underpinnings of LTV SSMs and leveraging control theory to enhance explainability and performance.This paper explores the integration of linear state-space models (SSMs) into deep neural network architectures, particularly in the context of foundation models like GPT-4. SSMs, which are traditionally used in control theory to model dynamical systems, offer a recurrent nature that captures past inputs efficiently, making them suitable for long-context tasks. The authors provide a gentle introduction to SSM-based architectures from a control theoretical perspective, reviewing the essential components and recent advancements. They highlight the mathematical structure, computational considerations, and initialization strategies of SSMs, emphasizing the importance of memory and eigenvalues in achieving long-range memory. The paper also presents a comparative analysis of various SSM proposals, including S4, S4D, S5, LRU, S6, and RG-LRU, on the Long Range Arena (LRA) benchmark, which evaluates models' reasoning ability and handling of diverse data types. The results show that LTI-based SSMs outperform LTV-based models, raising questions about the theoretical properties and eigenvalue significance in LTV models. The authors conclude by discussing future research opportunities, particularly in understanding the theoretical underpinnings of LTV SSMs and leveraging control theory to enhance explainability and performance.