22 May 2024 | George Shaikovski, Adam Casson, Kristen Severson, Eric Zimmermann, Yi Kan Wang, Jeremy D. Kunz, Juan A. Retamero, Gerard Oakley, David Klimstra, Christopher Kanan, Matthew Hanna, Michal Zelechowski, Julian Viret, Neil Tenenholtz, James Hall, Nicolò Fusi, Razik Yousfi, Peter Hamilton, William A. Moye, Eugene Vorontsov, Siqi Liu, Thomas J. Fuchs
PRISM is a multi-modal generative foundation model for slide-level histopathology. It is trained using clinical report data as supervisory signals and leverages Virchow tile embeddings for pre-training. PRISM produces slide-level embeddings with the ability to generate clinical reports, enabling several modes of use, including zero-shot cancer detection and sub-typing, and slide-level linear classification. It also demonstrates label-efficient training for biomarker prediction, outperforming supervised baselines with as little as 10% of the training data. PRISM is pre-trained on 587,000 WSIs and 195,000 clinical text reports. The model can generate text-based diagnosis reports for WSIs and accurately identify various WSI features, match slides to correct prompts, or generate text captions describing features without further training. PRISM is evaluated on cancer detection, tissue sub-typing, and biomarker prediction tasks using zero-shot classification, linear probing, and fine-tuning. It outperforms supervised baselines in these tasks. PRISM also demonstrates label-efficient training for biomarker prediction, with pre-training leading to more accurate results than training from scratch. The model's ability to generate reports allows for training-free zero-shot classification and enhances interpretability. PRISM is trained on diverse pathology reports and demonstrates the effectiveness of making predictions on a WSI level. Aligning to rich pathology reports boosts in-domain transfer performance and benefits transfer to tasks not covered by the reports. PRISM is a slide-level foundation model that can be used for a wide range of clinically-relevant tasks. The model is trained using a memory-efficient Perceiver network and a language decoder based on BioGPT. The training objective includes contrastive and generative objectives. PRISM is evaluated on cancer detection, tissue sub-typing, and biomarker prediction tasks, showing superior performance compared to supervised baselines. The model's ability to generate reports allows for training-free zero-shot classification and enhances interpretability. PRISM is a slide-level foundation model that can be used for a wide range of clinically-relevant tasks. The model is trained using a memory-efficient Perceiver network and a language decoder based on BioGPT. The training objective includes contrastive and generative objectives. PRISM is evaluated on cancer detection, tissue sub-typing, and biomarker prediction tasks, showing superior performance compared to supervised baselines.PRISM is a multi-modal generative foundation model for slide-level histopathology. It is trained using clinical report data as supervisory signals and leverages Virchow tile embeddings for pre-training. PRISM produces slide-level embeddings with the ability to generate clinical reports, enabling several modes of use, including zero-shot cancer detection and sub-typing, and slide-level linear classification. It also demonstrates label-efficient training for biomarker prediction, outperforming supervised baselines with as little as 10% of the training data. PRISM is pre-trained on 587,000 WSIs and 195,000 clinical text reports. The model can generate text-based diagnosis reports for WSIs and accurately identify various WSI features, match slides to correct prompts, or generate text captions describing features without further training. PRISM is evaluated on cancer detection, tissue sub-typing, and biomarker prediction tasks using zero-shot classification, linear probing, and fine-tuning. It outperforms supervised baselines in these tasks. PRISM also demonstrates label-efficient training for biomarker prediction, with pre-training leading to more accurate results than training from scratch. The model's ability to generate reports allows for training-free zero-shot classification and enhances interpretability. PRISM is trained on diverse pathology reports and demonstrates the effectiveness of making predictions on a WSI level. Aligning to rich pathology reports boosts in-domain transfer performance and benefits transfer to tasks not covered by the reports. PRISM is a slide-level foundation model that can be used for a wide range of clinically-relevant tasks. The model is trained using a memory-efficient Perceiver network and a language decoder based on BioGPT. The training objective includes contrastive and generative objectives. PRISM is evaluated on cancer detection, tissue sub-typing, and biomarker prediction tasks, showing superior performance compared to supervised baselines. The model's ability to generate reports allows for training-free zero-shot classification and enhances interpretability. PRISM is a slide-level foundation model that can be used for a wide range of clinically-relevant tasks. The model is trained using a memory-efficient Perceiver network and a language decoder based on BioGPT. The training objective includes contrastive and generative objectives. PRISM is evaluated on cancer detection, tissue sub-typing, and biomarker prediction tasks, showing superior performance compared to supervised baselines.