10 Mar 2024 | Yuxuan Bian, Xuan Ju, Jiangtong Li, Zhijian Xu, Dawei Cheng, Qiang Xu
This paper introduces aLLM4TS, a novel framework that adapts Large Language Models (LLMs) for time-series representation learning. The key idea is to reframe time-series forecasting as a self-supervised, multi-patch prediction task, which more effectively captures temporal dynamics in patch representations compared to traditional contrastive learning or mask-and-reconstruction methods. The framework consists of two stages: (1) causal continual pre-training on various time-series datasets, focusing on next-patch prediction to align LLM capabilities with time-series data intricacies; and (2) fine-tuning for multi-patch prediction in targeted time-series contexts. A distinctive feature is the patch-wise decoding layer, which decodes each patch independently into temporal sequences, enhancing the model's ability to learn temporal patch-based representations. aLLM4TS demonstrates superior performance in various downstream tasks, showing its effectiveness in deriving temporal representations with enhanced transferability. The framework also addresses the limitations of traditional sequence-wise decoding by using a patch-wise decoder, which avoids temporal dependencies disruption and aligns with LLM pre-training processes. The results show that aLLM4TS outperforms existing methods in long-term and short-term forecasting, few-shot forecasting, and anomaly detection tasks. The model's performance is validated through extensive experiments on multiple datasets, demonstrating its robustness and adaptability across different time-series domains. The study highlights the potential of LLMs in time-series analysis and provides a new approach for adapting LLMs to time-series representation learning.This paper introduces aLLM4TS, a novel framework that adapts Large Language Models (LLMs) for time-series representation learning. The key idea is to reframe time-series forecasting as a self-supervised, multi-patch prediction task, which more effectively captures temporal dynamics in patch representations compared to traditional contrastive learning or mask-and-reconstruction methods. The framework consists of two stages: (1) causal continual pre-training on various time-series datasets, focusing on next-patch prediction to align LLM capabilities with time-series data intricacies; and (2) fine-tuning for multi-patch prediction in targeted time-series contexts. A distinctive feature is the patch-wise decoding layer, which decodes each patch independently into temporal sequences, enhancing the model's ability to learn temporal patch-based representations. aLLM4TS demonstrates superior performance in various downstream tasks, showing its effectiveness in deriving temporal representations with enhanced transferability. The framework also addresses the limitations of traditional sequence-wise decoding by using a patch-wise decoder, which avoids temporal dependencies disruption and aligns with LLM pre-training processes. The results show that aLLM4TS outperforms existing methods in long-term and short-term forecasting, few-shot forecasting, and anomaly detection tasks. The model's performance is validated through extensive experiments on multiple datasets, demonstrating its robustness and adaptability across different time-series domains. The study highlights the potential of LLMs in time-series analysis and provides a new approach for adapting LLMs to time-series representation learning.