This paper introduces GPT4MTS, a prompt-based large language model for multimodal time series forecasting. The authors propose a general principle for collecting textual information from various sources using modern large language models (LLMs). They then develop a prompt-based LLM framework that integrates both numerical data and textual information, named GPT4MTS. The paper presents a GDELT-based multimodal time series dataset for news impact forecasting, which includes both numerical time series data and textual summaries of events. Through extensive experiments, the authors demonstrate the effectiveness of their method in forecasting tasks with extra-textual information. Time series forecasting is essential in various fields, including finance, economics, healthcare, and weather prediction. Most previous forecasting models focus on unimodal numerical data, but the integration of textual information can enhance forecasting performance. However, collecting and fusing multimodal information is challenging. The authors propose a pipeline that leverages LLMs to generate textual data alongside time series data, including textual information collection, summarization, re-ranking, and efficient summary based on re-ranking similarity. The authors create a GDELT-based multimodal time series forecasting dataset that includes both numerical values and textual summaries of events. This dataset is derived from the GDELT database, which records global events and their associated media coverage. The dataset is used to enhance the accessibility of multimodal time series datasets and to foster further research in computational communication analysis. The authors propose a prompt tuning-based LLM, GPT4MTS, for time series forecasting with multimodal input. This model contrasts with conventional approaches that rely on direct data alignment. For numerical input, the model splits the temporal input into different patches and applies a linear layer to embed the patches into a hidden space as the time series input embedding. For textual information, the model uses pre-trained language models to obtain textual embeddings and treats them as trainable soft prompts prepended to the temporal input. The model also freezes the attention layer inside the LLM to speed up training and inference. The authors' experiments show that their model achieves better performance than previous models, with a 4.14% reduction in MSE and a 1.0% reduction in MAE. The model's performance is attributed to the integration of numerical data, textual information, and a prompt-based methodology for multimodal integration. The authors also discuss the relevance of their approach for enhancing communication accessibility, particularly for individuals in underdeveloped regions, those with reading disabilities, and non-English speakers. The model's ability to provide summarized news and key information through numerical data enhances the global news reach and benefits non-English speakers and minority language users. The authors conclude that their approach is effective for multimodal time series forecasting and has potential for future research in this area.This paper introduces GPT4MTS, a prompt-based large language model for multimodal time series forecasting. The authors propose a general principle for collecting textual information from various sources using modern large language models (LLMs). They then develop a prompt-based LLM framework that integrates both numerical data and textual information, named GPT4MTS. The paper presents a GDELT-based multimodal time series dataset for news impact forecasting, which includes both numerical time series data and textual summaries of events. Through extensive experiments, the authors demonstrate the effectiveness of their method in forecasting tasks with extra-textual information. Time series forecasting is essential in various fields, including finance, economics, healthcare, and weather prediction. Most previous forecasting models focus on unimodal numerical data, but the integration of textual information can enhance forecasting performance. However, collecting and fusing multimodal information is challenging. The authors propose a pipeline that leverages LLMs to generate textual data alongside time series data, including textual information collection, summarization, re-ranking, and efficient summary based on re-ranking similarity. The authors create a GDELT-based multimodal time series forecasting dataset that includes both numerical values and textual summaries of events. This dataset is derived from the GDELT database, which records global events and their associated media coverage. The dataset is used to enhance the accessibility of multimodal time series datasets and to foster further research in computational communication analysis. The authors propose a prompt tuning-based LLM, GPT4MTS, for time series forecasting with multimodal input. This model contrasts with conventional approaches that rely on direct data alignment. For numerical input, the model splits the temporal input into different patches and applies a linear layer to embed the patches into a hidden space as the time series input embedding. For textual information, the model uses pre-trained language models to obtain textual embeddings and treats them as trainable soft prompts prepended to the temporal input. The model also freezes the attention layer inside the LLM to speed up training and inference. The authors' experiments show that their model achieves better performance than previous models, with a 4.14% reduction in MSE and a 1.0% reduction in MAE. The model's performance is attributed to the integration of numerical data, textual information, and a prompt-based methodology for multimodal integration. The authors also discuss the relevance of their approach for enhancing communication accessibility, particularly for individuals in underdeveloped regions, those with reading disabilities, and non-English speakers. The model's ability to provide summarized news and key information through numerical data enhances the global news reach and benefits non-English speakers and minority language users. The authors conclude that their approach is effective for multimodal time series forecasting and has potential for future research in this area.