**SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection** **Peng Qi, Zehong Yan, Wynne Hsu, Mong Li Lee** **National University of Singapore** **Abstract:** Misinformation is a significant societal issue, and out-of-context (OOC) misinformation, where authentic images are repurposed with false text, is one of the most effective ways to mislead audiences. Current methods focus on assessing image-text consistency but lack convincing explanations for their judgments. This paper introduces SNIFFER, a novel multimodal large language model specifically engineered for OOC misinformation detection and explanation. SNIFFER employs a two-stage instruction tuning process on InstructBLIP, refining the model's concept alignment of generic objects with news-domain entities and leveraging language-only GPT-4 generated OOC-specific instruction data to enhance its discriminatory powers. Enhanced by external tools and retrieval, SNIFFER not only detects inconsistencies between text and image but also utilizes external knowledge for contextual verification. Experiments show that SNIFFER surpasses the original MLLM by over 40% and outperforms state-of-the-art methods in detection accuracy. SNIFFER also provides accurate and persuasive explanations, validated by quantitative and human evaluations. **Introduction:** OOC misinformation is a prevalent issue due to its potential high risks. Current methods focus on assessing image-text consistency but lack convincing explanations. Multimodal large language models (MLLMs) have rich knowledge and visual reasoning capabilities but lack sophistication in understanding cross-modal differences. SNIFFER addresses this by employing a two-stage instruction tuning process and integrating external knowledge through retrieval and tool usage. It conducts internal and external checks to identify inconsistencies and reason between the given text and retrieved image context, integrating these into a unified output. **Method:** SNIFFER uses a two-stage instruction tuning procedure on InstructBLIP, refining concept alignment and leveraging OOC-specific data. It integrates external knowledge through retrieval and tool usage, enhancing its ability to detect and explain OOC misinformation. **Performance Study:** Experiments show that SNIFFER outperforms baselines in detection accuracy and generates accurate and persuasive explanations. Ablation studies demonstrate the importance of each component in SNIFFER's performance. Human evaluations confirm the effectiveness of SNIFFER's explanations in debunking misinformation. **Conclusion:** SNIFFER is a novel multimodal large language model for OOC misinformation detection, providing both judgment and explanation. It achieves state-of-the-art performance and generates precise and persuasive explanations, making it a valuable tool for combating misinformation.**SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection** **Peng Qi, Zehong Yan, Wynne Hsu, Mong Li Lee** **National University of Singapore** **Abstract:** Misinformation is a significant societal issue, and out-of-context (OOC) misinformation, where authentic images are repurposed with false text, is one of the most effective ways to mislead audiences. Current methods focus on assessing image-text consistency but lack convincing explanations for their judgments. This paper introduces SNIFFER, a novel multimodal large language model specifically engineered for OOC misinformation detection and explanation. SNIFFER employs a two-stage instruction tuning process on InstructBLIP, refining the model's concept alignment of generic objects with news-domain entities and leveraging language-only GPT-4 generated OOC-specific instruction data to enhance its discriminatory powers. Enhanced by external tools and retrieval, SNIFFER not only detects inconsistencies between text and image but also utilizes external knowledge for contextual verification. Experiments show that SNIFFER surpasses the original MLLM by over 40% and outperforms state-of-the-art methods in detection accuracy. SNIFFER also provides accurate and persuasive explanations, validated by quantitative and human evaluations. **Introduction:** OOC misinformation is a prevalent issue due to its potential high risks. Current methods focus on assessing image-text consistency but lack convincing explanations. Multimodal large language models (MLLMs) have rich knowledge and visual reasoning capabilities but lack sophistication in understanding cross-modal differences. SNIFFER addresses this by employing a two-stage instruction tuning process and integrating external knowledge through retrieval and tool usage. It conducts internal and external checks to identify inconsistencies and reason between the given text and retrieved image context, integrating these into a unified output. **Method:** SNIFFER uses a two-stage instruction tuning procedure on InstructBLIP, refining concept alignment and leveraging OOC-specific data. It integrates external knowledge through retrieval and tool usage, enhancing its ability to detect and explain OOC misinformation. **Performance Study:** Experiments show that SNIFFER outperforms baselines in detection accuracy and generates accurate and persuasive explanations. Ablation studies demonstrate the importance of each component in SNIFFER's performance. Human evaluations confirm the effectiveness of SNIFFER's explanations in debunking misinformation. **Conclusion:** SNIFFER is a novel multimodal large language model for OOC misinformation detection, providing both judgment and explanation. It achieves state-of-the-art performance and generates precise and persuasive explanations, making it a valuable tool for combating misinformation.