This paper investigates whether the Mamba architecture, a state space model, can perform in-context learning (ICL), a form of meta-learning where a model solves tasks based on input examples without explicit training. The study compares Mamba with transformer models, which are currently the state of the art in ICL. The results show that Mamba performs similarly to transformers on both simple function approximation tasks and complex natural language processing (NLP) tasks. Mamba also outperforms its predecessor, S4, and a recent parallel/recurrent architecture, RWKV. The study finds that Mamba incrementally optimizes its internal representations to solve ICL tasks, similar to how transformers do. This suggests that Mamba can be an efficient alternative to transformers for ICL tasks involving long input sequences. The findings have implications for generalizing in-context learned AutoML algorithms to long input sequences. The code to reproduce the experiments is available at github.com/automl/is_mamba_capable_of_icl. The paper also explores the performance of Mamba on various NLP tasks, showing that it achieves comparable or better results than other models, including transformer-based models like LLama, Pythia, and GPT-J. Mamba scales well with the number of in-context examples and maintains a performance advantage over RWKV in different model sizes. The study highlights the potential of Mamba as a scalable and efficient alternative to transformers for ICL tasks. It also identifies areas for future research, including a deeper understanding of Mamba's mechanisms and its performance across different domains. The findings contribute to the broader understanding of ICL and its applications in AI systems.This paper investigates whether the Mamba architecture, a state space model, can perform in-context learning (ICL), a form of meta-learning where a model solves tasks based on input examples without explicit training. The study compares Mamba with transformer models, which are currently the state of the art in ICL. The results show that Mamba performs similarly to transformers on both simple function approximation tasks and complex natural language processing (NLP) tasks. Mamba also outperforms its predecessor, S4, and a recent parallel/recurrent architecture, RWKV. The study finds that Mamba incrementally optimizes its internal representations to solve ICL tasks, similar to how transformers do. This suggests that Mamba can be an efficient alternative to transformers for ICL tasks involving long input sequences. The findings have implications for generalizing in-context learned AutoML algorithms to long input sequences. The code to reproduce the experiments is available at github.com/automl/is_mamba_capable_of_icl. The paper also explores the performance of Mamba on various NLP tasks, showing that it achieves comparable or better results than other models, including transformer-based models like LLama, Pythia, and GPT-J. Mamba scales well with the number of in-context examples and maintains a performance advantage over RWKV in different model sizes. The study highlights the potential of Mamba as a scalable and efficient alternative to transformers for ICL tasks. It also identifies areas for future research, including a deeper understanding of Mamba's mechanisms and its performance across different domains. The findings contribute to the broader understanding of ICL and its applications in AI systems.