HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images Simon Graham, Quoc Dang Vu, Shan E Ahmed Raza, Ayesha Azam, Yee Wah Tsang, Jin Tae Kwak, and Nasir Rajpoot Abstract: Nuclear segmentation and classification in H&E stained histology images is a fundamental step in digital pathology. Automated methods enable quantitative analysis of thousands of nuclei in whole-slide images, opening possibilities for large-scale nuclear morphometry analysis. However, automated segmentation and classification face challenges due to varying nuclear types and clustering. HoVer-Net is a novel CNN that uses horizontal and vertical distances from nuclear pixels to their centers to separate clustered nuclei. It predicts nuclear types via an up-sampling branch. HoVer-Net achieves state-of-the-art performance on multiple datasets and introduces a new colorectal adenocarcinoma dataset with 24,319 annotated nuclei. Keywords: Nuclear segmentation, nuclear classification, computational pathology, deep learning Introduction: Manual assessment of H&E stained slides is slow and prone to variability. Digital pathology offers efficient processing of whole-slide images. Nuclear segmentation and classification are crucial for clinical outcomes. However, nuclei display high heterogeneity and clustering, making segmentation and classification challenging. Current methods use separate models for segmentation and classification. HoVer-Net proposes a unified model for simultaneous segmentation and classification. Related Work: Energy-based methods like watershed have been used for nuclear segmentation. Deep learning methods like U-Net and FCN have been applied to medical image analysis. Recent methods like Micro-Net and CIA-Net have been proposed for instance segmentation. Nuclear classification is typically done via two-stage approaches. HoVer-Net introduces a novel approach for simultaneous segmentation and classification. Methods: HoVer-Net uses a pre-activated ResNet50 encoder and three up-sampling branches: NP, HoVer, and NC. The NP branch predicts nuclear pixels, the HoVer branch predicts horizontal and vertical distances, and the NC branch predicts nuclear types. The network uses dense units and skip connections for efficient gradient propagation. The loss function combines regression and classification losses. Post-processing uses Sobel gradients to separate clustered nuclei. Evaluation: HoVer-Net is evaluated on six datasets, including a new colorectal adenocarcinoma dataset. Metrics include DICE, AJI, and Panoptic Quality (PQ). PQ is proposed as a more reliable metric for instance segmentation. HoVer-Net achieves state-of-the-art performance on all datasets. Results: HoVer-Net outperforms existing methods in segmentation and classification. It achieves high PQ scores, indicating accurate detection and segmentation. The model performs well on challenging datasets with overlapping nuclei. The network is robust to staining variations and can handle different nuclear types. Conclusion: HoVer-Net is a novel deep learning approach for simultaneous nuclear segmentation and classification in histology images. It achieves state-ofHoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images Simon Graham, Quoc Dang Vu, Shan E Ahmed Raza, Ayesha Azam, Yee Wah Tsang, Jin Tae Kwak, and Nasir Rajpoot Abstract: Nuclear segmentation and classification in H&E stained histology images is a fundamental step in digital pathology. Automated methods enable quantitative analysis of thousands of nuclei in whole-slide images, opening possibilities for large-scale nuclear morphometry analysis. However, automated segmentation and classification face challenges due to varying nuclear types and clustering. HoVer-Net is a novel CNN that uses horizontal and vertical distances from nuclear pixels to their centers to separate clustered nuclei. It predicts nuclear types via an up-sampling branch. HoVer-Net achieves state-of-the-art performance on multiple datasets and introduces a new colorectal adenocarcinoma dataset with 24,319 annotated nuclei. Keywords: Nuclear segmentation, nuclear classification, computational pathology, deep learning Introduction: Manual assessment of H&E stained slides is slow and prone to variability. Digital pathology offers efficient processing of whole-slide images. Nuclear segmentation and classification are crucial for clinical outcomes. However, nuclei display high heterogeneity and clustering, making segmentation and classification challenging. Current methods use separate models for segmentation and classification. HoVer-Net proposes a unified model for simultaneous segmentation and classification. Related Work: Energy-based methods like watershed have been used for nuclear segmentation. Deep learning methods like U-Net and FCN have been applied to medical image analysis. Recent methods like Micro-Net and CIA-Net have been proposed for instance segmentation. Nuclear classification is typically done via two-stage approaches. HoVer-Net introduces a novel approach for simultaneous segmentation and classification. Methods: HoVer-Net uses a pre-activated ResNet50 encoder and three up-sampling branches: NP, HoVer, and NC. The NP branch predicts nuclear pixels, the HoVer branch predicts horizontal and vertical distances, and the NC branch predicts nuclear types. The network uses dense units and skip connections for efficient gradient propagation. The loss function combines regression and classification losses. Post-processing uses Sobel gradients to separate clustered nuclei. Evaluation: HoVer-Net is evaluated on six datasets, including a new colorectal adenocarcinoma dataset. Metrics include DICE, AJI, and Panoptic Quality (PQ). PQ is proposed as a more reliable metric for instance segmentation. HoVer-Net achieves state-of-the-art performance on all datasets. Results: HoVer-Net outperforms existing methods in segmentation and classification. It achieves high PQ scores, indicating accurate detection and segmentation. The model performs well on challenging datasets with overlapping nuclei. The network is robust to staining variations and can handle different nuclear types. Conclusion: HoVer-Net is a novel deep learning approach for simultaneous nuclear segmentation and classification in histology images. It achieves state-of