Are Transformers Effective for Time Series Forecasting?

Are Transformers Effective for Time Series Forecasting?

17 Aug 2022 | Ailing Zeng, Muxi Chen, Lei Zhang, Qiang Xu
This paper questions the effectiveness of Transformer-based models for long-term time series forecasting (LTSE). Despite their success in various sequence modeling tasks, Transformers are argued to be less suitable for LTSE due to their permutation-invariant self-attention mechanism, which can lead to temporal information loss. To validate this claim, the authors introduce a simple one-layer linear model named LTSF-Linear and compare it with existing Transformer-based models on nine real-life datasets. The results show that LTSF-Linear outperforms existing Transformer-based models in all cases, often by a significant margin. The study also explores the impact of various design elements in Transformer-based models and concludes that the temporal modeling capabilities of Transformers for time series are exaggerated. The authors advocate for revisiting the validity of Transformer-based solutions for other time series analysis tasks and provide a new baseline, LTSF-Linear, for future research.This paper questions the effectiveness of Transformer-based models for long-term time series forecasting (LTSE). Despite their success in various sequence modeling tasks, Transformers are argued to be less suitable for LTSE due to their permutation-invariant self-attention mechanism, which can lead to temporal information loss. To validate this claim, the authors introduce a simple one-layer linear model named LTSF-Linear and compare it with existing Transformer-based models on nine real-life datasets. The results show that LTSF-Linear outperforms existing Transformer-based models in all cases, often by a significant margin. The study also explores the impact of various design elements in Transformer-based models and concludes that the temporal modeling capabilities of Transformers for time series are exaggerated. The authors advocate for revisiting the validity of Transformer-based solutions for other time series analysis tasks and provide a new baseline, LTSF-Linear, for future research.
Reach us at info@study.space
[slides and audio] Are Transformers Effective for Time Series Forecasting%3F