**FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models** Gagan Bhatia, El Moatez Billah Nagoudi, Hasan Cavusoglu, Muhammad Abdul-Mageed The University of British Columbia & Invertible AI {gagan30@student., moatez.nagoudi@, cavusoglu@sauder}ubc.ca {muhammad.mageed@}jubc.ca; invertible.ai ## Abstract We introduce *FinTral*, a suite of state-of-the-art multimodal large language models (LLMs) built upon the Mistral-7b model and tailored for financial analysis. FinTral integrates textual, numerical, tabular, and image data. We present *FinSet*, the largest financial LLM pre-training, instruction tuning, and financial alignment dataset and evaluation benchmark featuring nine tasks and 23 datasets, and the first to understand and mitigate financial hallucinations. We enhance *FinTral* with domain-specific pretraining, instruction fine-tuning, and RLAIF training by exploiting a large collection of textual and visual datasets we curate for this work. Our FinTral model trained with direct preference optimization employing advanced Tools and Retrieval methods, dubbed *FinTral-DPO-T&R*, demonstrates an exceptional zero-shot performance. It outperforms ChatGPT-3.5 in all tasks and surpasses GPT-4 in five out of nine tasks, marking a significant advancement in AI-driven financial technology. We also demonstrate that FinTral has the potential to excel in real-time analysis and decision-making in diverse financial contexts. The GitHub repository for *FinTral* is available at <https://github.com/UBC-NLP/fintral>. Natural Language Processing (NLP) plays a key role in financial document analysis, interpretation, and utilization. In recent years, a wide range of applications incorporating advances in NLP have emerged. These include sentiment analysis of financial news, event extraction from financial documents, and the generation and summarization of financial reports. However, applying NLP in finance is challenging due to the complexity of financial language, the scarcity of annotated data, and the need for real-time analysis. Financial documents often include dense numerical information and domain-specific jargon, requiring advanced numerical processing and reasoning capabilities. This necessitates extensive domain knowledge for models to capture the nuanced implications of accounting and financial measures, economic indicators, and market trends. To meet these challenges, we introduce a groundbreaking LLM specialized in the financial domain. Our model, dubbed *FinTral*, is designed to overcome hurdles in the financial domain through a multimodal approach that integrates textual, numerical, tabular, and visual data processing for comprehensive document understanding. We train our model off Mistral-7b on a sizeable domain-specific dataset and instruction-tune it for the financial domain using extensive instruction data.**FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models** Gagan Bhatia, El Moatez Billah Nagoudi, Hasan Cavusoglu, Muhammad Abdul-Mageed The University of British Columbia & Invertible AI {gagan30@student., moatez.nagoudi@, cavusoglu@sauder}ubc.ca {muhammad.mageed@}jubc.ca; invertible.ai ## Abstract We introduce *FinTral*, a suite of state-of-the-art multimodal large language models (LLMs) built upon the Mistral-7b model and tailored for financial analysis. FinTral integrates textual, numerical, tabular, and image data. We present *FinSet*, the largest financial LLM pre-training, instruction tuning, and financial alignment dataset and evaluation benchmark featuring nine tasks and 23 datasets, and the first to understand and mitigate financial hallucinations. We enhance *FinTral* with domain-specific pretraining, instruction fine-tuning, and RLAIF training by exploiting a large collection of textual and visual datasets we curate for this work. Our FinTral model trained with direct preference optimization employing advanced Tools and Retrieval methods, dubbed *FinTral-DPO-T&R*, demonstrates an exceptional zero-shot performance. It outperforms ChatGPT-3.5 in all tasks and surpasses GPT-4 in five out of nine tasks, marking a significant advancement in AI-driven financial technology. We also demonstrate that FinTral has the potential to excel in real-time analysis and decision-making in diverse financial contexts. The GitHub repository for *FinTral* is available at <https://github.com/UBC-NLP/fintral>. Natural Language Processing (NLP) plays a key role in financial document analysis, interpretation, and utilization. In recent years, a wide range of applications incorporating advances in NLP have emerged. These include sentiment analysis of financial news, event extraction from financial documents, and the generation and summarization of financial reports. However, applying NLP in finance is challenging due to the complexity of financial language, the scarcity of annotated data, and the need for real-time analysis. Financial documents often include dense numerical information and domain-specific jargon, requiring advanced numerical processing and reasoning capabilities. This necessitates extensive domain knowledge for models to capture the nuanced implications of accounting and financial measures, economic indicators, and market trends. To meet these challenges, we introduce a groundbreaking LLM specialized in the financial domain. Our model, dubbed *FinTral*, is designed to overcome hurdles in the financial domain through a multimodal approach that integrates textual, numerical, tabular, and visual data processing for comprehensive document understanding. We train our model off Mistral-7b on a sizeable domain-specific dataset and instruction-tune it for the financial domain using extensive instruction data.