The paper introduces ContiFormer, a novel continuous-time Transformer model designed to handle irregular time series data. Traditional methods like recurrent neural networks (RNNs) and Transformers struggle with the discrete nature of time series data, while Neural Ordinary Differential Equations (Neural ODEs) and their variants often fail to capture intricate correlations within these sequences. ContiFormer addresses these challenges by extending the Transformer's attention mechanism to the continuous-time domain, incorporating the continuous dynamics of Neural ODEs. The model is mathematically characterized, showing that various Transformer variants can be seen as special cases of ContiFormer. Extensive experiments on synthetic and real-world datasets demonstrate ContiFormer's superior performance in modeling complex continuous-time dynamic systems, outperforming both RNN-based and Transformer-based methods in tasks such as interpolation, extrapolation, classification, event prediction, and forecasting. The paper also provides a detailed analysis of the model's complexity and efficiency, highlighting its competitive performance and robustness to step size.The paper introduces ContiFormer, a novel continuous-time Transformer model designed to handle irregular time series data. Traditional methods like recurrent neural networks (RNNs) and Transformers struggle with the discrete nature of time series data, while Neural Ordinary Differential Equations (Neural ODEs) and their variants often fail to capture intricate correlations within these sequences. ContiFormer addresses these challenges by extending the Transformer's attention mechanism to the continuous-time domain, incorporating the continuous dynamics of Neural ODEs. The model is mathematically characterized, showing that various Transformer variants can be seen as special cases of ContiFormer. Extensive experiments on synthetic and real-world datasets demonstrate ContiFormer's superior performance in modeling complex continuous-time dynamic systems, outperforming both RNN-based and Transformer-based methods in tasks such as interpolation, extrapolation, classification, event prediction, and forecasting. The paper also provides a detailed analysis of the model's complexity and efficiency, highlighting its competitive performance and robustness to step size.