MAIRA-2: Grounded Radiology Report Generation

MAIRA-2: Grounded Radiology Report Generation

20 Sep 2024 | Shruthi Bannur, Kenza Bouzid, Daniel C. Castro, Anton Schwaighofer, Anja Thieme, Sam Bond-Taylor, Maximilian Ilse, Fernando Pérez-García, Valentina Salvatelli, Harshita Sharma, Felix Meissen, Mercy Ranjit, Shaury Srivastav, Julia Gong, Noel C. F. Codella, Fabian Falck, Ozan Oktay, Matthew P. Lungren, Maria Teodora Wetscherek, Javier Alvarez-Valle, Stephanie L. Hyland
MAIRA-2 is a large radiology-specific multimodal model designed to generate chest X-ray reports with and without grounding. The model incorporates spatial localization of findings on images, a task known as grounded report generation, and enhances performance by incorporating realistic reporting context. MAIRA-2 achieves state-of-the-art results on existing report generation benchmarks and introduces the novel task of grounded report generation. The paper presents a new evaluation framework, RadFact, which leverages large language models (LLMs) to assess the correctness and completeness of generated reports at the sentence level. RadFact supports the evaluation of grounded reporting by analyzing the logical entailment of generated sentences against reference ground truth. The model is trained on a diverse set of public and private chest X-ray datasets, including MIMIC-CXR, Pad-Chest, and USMix. MAIRA-2 is capable of generating both grounded and non-grounded reports, integrating additional inputs such as lateral views, prior studies, and clinical information. The model's architecture includes a specialized image encoder and a language model trained on a multitask basis. The paper also discusses the importance of incorporating additional context and spatial annotations in radiology report generation, highlighting the benefits of grounded reporting for verification, image comprehension, and new use cases in generalist medical AI. The study demonstrates that MAIRA-2 outperforms previous approaches in findings generation and establishes a new state-of-the-art in non-grounded report generation. The evaluation framework RadFact provides a fine-grained assessment of model performance, capturing aspects of precision and recall at both text-only and text-and-grounding levels. The results show that MAIRA-2 achieves high logical precision and recall on grounded report generation, indicating its ability to accurately describe and localize findings. The model also performs well on non-grounded report generation, achieving significant improvements in lexical metrics. The paper concludes that grounded radiology report generation is a promising task that can enhance the accuracy and reliability of automated report generation, with MAIRA-2 serving as a baseline for future research in this area.MAIRA-2 is a large radiology-specific multimodal model designed to generate chest X-ray reports with and without grounding. The model incorporates spatial localization of findings on images, a task known as grounded report generation, and enhances performance by incorporating realistic reporting context. MAIRA-2 achieves state-of-the-art results on existing report generation benchmarks and introduces the novel task of grounded report generation. The paper presents a new evaluation framework, RadFact, which leverages large language models (LLMs) to assess the correctness and completeness of generated reports at the sentence level. RadFact supports the evaluation of grounded reporting by analyzing the logical entailment of generated sentences against reference ground truth. The model is trained on a diverse set of public and private chest X-ray datasets, including MIMIC-CXR, Pad-Chest, and USMix. MAIRA-2 is capable of generating both grounded and non-grounded reports, integrating additional inputs such as lateral views, prior studies, and clinical information. The model's architecture includes a specialized image encoder and a language model trained on a multitask basis. The paper also discusses the importance of incorporating additional context and spatial annotations in radiology report generation, highlighting the benefits of grounded reporting for verification, image comprehension, and new use cases in generalist medical AI. The study demonstrates that MAIRA-2 outperforms previous approaches in findings generation and establishes a new state-of-the-art in non-grounded report generation. The evaluation framework RadFact provides a fine-grained assessment of model performance, capturing aspects of precision and recall at both text-only and text-and-grounding levels. The results show that MAIRA-2 achieves high logical precision and recall on grounded report generation, indicating its ability to accurately describe and localize findings. The model also performs well on non-grounded report generation, achieving significant improvements in lexical metrics. The paper concludes that grounded radiology report generation is a promising task that can enhance the accuracy and reliability of automated report generation, with MAIRA-2 serving as a baseline for future research in this area.
Reach us at info@study.space
[slides and audio] MAIRA-2%3A Grounded Radiology Report Generation