HistGen: Histopathology Report Generation via Local-Global Feature Encoding and Cross-modal Context Interaction

HistGen: Histopathology Report Generation via Local-Global Feature Encoding and Cross-modal Context Interaction

18 Jun 2024 | Zhengrui Guo, Jiabao Ma, Yingxue Xu, Yihui Wang, Liansheng Wang, Hao Chen
HistGen is a framework for automated histopathology report generation, combining local and global feature encoding with cross-modal context interaction. The framework is designed to align whole slide images (WSIs) and diagnostic reports at both local and global levels. It features a local-global hierarchical encoder for efficient visual feature aggregation and a cross-modal context module to facilitate alignment between visual and textual modalities. The model is pre-trained on over 55,000 WSIs to enhance feature encoding and is evaluated on a benchmark dataset of around 7,800 WSI-report pairs. Experimental results show that HistGen outperforms state-of-the-art models in WSI report generation and demonstrates strong transfer learning capabilities in cancer subtyping and survival analysis tasks. The framework also includes a transfer learning strategy for cancer diagnosis and prognosis. HistGen addresses the challenges of WSI report generation, including the lack of benchmark datasets, the gigapixel size of WSIs, and the need for cross-modal interactions. The model's performance is validated through extensive experiments, showing its effectiveness in generating accurate and meaningful histopathology reports.HistGen is a framework for automated histopathology report generation, combining local and global feature encoding with cross-modal context interaction. The framework is designed to align whole slide images (WSIs) and diagnostic reports at both local and global levels. It features a local-global hierarchical encoder for efficient visual feature aggregation and a cross-modal context module to facilitate alignment between visual and textual modalities. The model is pre-trained on over 55,000 WSIs to enhance feature encoding and is evaluated on a benchmark dataset of around 7,800 WSI-report pairs. Experimental results show that HistGen outperforms state-of-the-art models in WSI report generation and demonstrates strong transfer learning capabilities in cancer subtyping and survival analysis tasks. The framework also includes a transfer learning strategy for cancer diagnosis and prognosis. HistGen addresses the challenges of WSI report generation, including the lack of benchmark datasets, the gigapixel size of WSIs, and the need for cross-modal interactions. The model's performance is validated through extensive experiments, showing its effectiveness in generating accurate and meaningful histopathology reports.
Reach us at info@study.space