Segment Any Cell: A SAM-based Auto-prompting Fine-tuning Framework for Nuclei Segmentation

Segment Any Cell: A SAM-based Auto-prompting Fine-tuning Framework for Nuclei Segmentation

24 Jan 2024 | Saiyang Na, Yuzhi Guo, Feng Jiang, Hehuan Ma, Junzhou Huang
Segment Any Cell (SAC) is an innovative auto-prompting fine-tuning framework for nuclei segmentation based on the Segment Anything Model (SAM). SAC enhances SAM specifically for nuclei segmentation by integrating Low-Rank Adaptation (LoRA) within the Transformer's attention layer, improving the fine-tuning process and outperforming existing state-of-the-art (SOTA) methods. It also introduces an auto-prompt generator that produces effective prompts to guide segmentation, a critical factor in handling the complexities of nuclei segmentation in biomedical imaging. SAC's contributions include a novel prompt generation strategy, automated adaptability for diverse segmentation tasks, the innovative application of Low-Rank Attention Adaptation in SAM, and a versatile framework for semantic segmentation challenges. SAC addresses the challenge of generating high-quality, informative prompts, which is as crucial as applying SOTA fine-tuning techniques on foundation models. The framework automatically generates prompts from medical images, enabling efficient and accurate nuclei segmentation. SAC's auto-prompt generator creates precise prompts from medical images to guide the segmentation process, overcoming the limitations of SAM in obtaining suitable prompts for nuclei segmentation tasks. Through extensive experiments, SAC demonstrates superior performance in nuclei segmentation tasks, proving its effectiveness as a tool for pathologists and researchers. The SAC framework includes a LoRA implementation within the SAM model to enhance the fine-tuning process, which outperforms existing fine-tuning/adapter-based SOTA methods. The auto-prompt generator is designed to automatically generate a large number of high-quality prompts to guide the segmentation process. This addresses the limitation of SAM's difficulty in obtaining suitable prompts for nuclei segmentation tasks. SAC's method is fully automated and easily adaptable to various nuclei segmentation tasks, offering a simplified yet effective tool for pathologists and researchers. SAC's auto-prompt generator is an auxiliary neural network that automates the process of generating prompts. It uses an auxiliary neural network to generate masks from input images, which are then used to create positive and negative prompts. These prompts are fed into the SAM prompt encoder to obtain prompt embeddings, which are then used in the SAM mask decoder to produce the final segmentation results. The auto-prompt generator also includes prompt discrimination capabilities, which are crucial for cell nuclei segmentation as SAM's ability to distinguish between positive and negative prompts significantly impacts segmentation accuracy. SAC's method is evaluated on two datasets: MoNuSeg and the 2018 Data Science Bowl (DSB). The results show that SAC outperforms existing SOTA methods in nuclei segmentation tasks, demonstrating its effectiveness and superiority. The method's performance is measured using F1 score, Dice coefficient, and Intersection over Union (IoU). SAC's auto-prompt generator and LoRA implementation significantly enhance the model's performance in nuclei segmentation tasks, making it a valuable tool for medical image analysis. The framework's ability to automatically generate prompts and adapt to various segmentation tasks highlights its generalizability and flexibility.Segment Any Cell (SAC) is an innovative auto-prompting fine-tuning framework for nuclei segmentation based on the Segment Anything Model (SAM). SAC enhances SAM specifically for nuclei segmentation by integrating Low-Rank Adaptation (LoRA) within the Transformer's attention layer, improving the fine-tuning process and outperforming existing state-of-the-art (SOTA) methods. It also introduces an auto-prompt generator that produces effective prompts to guide segmentation, a critical factor in handling the complexities of nuclei segmentation in biomedical imaging. SAC's contributions include a novel prompt generation strategy, automated adaptability for diverse segmentation tasks, the innovative application of Low-Rank Attention Adaptation in SAM, and a versatile framework for semantic segmentation challenges. SAC addresses the challenge of generating high-quality, informative prompts, which is as crucial as applying SOTA fine-tuning techniques on foundation models. The framework automatically generates prompts from medical images, enabling efficient and accurate nuclei segmentation. SAC's auto-prompt generator creates precise prompts from medical images to guide the segmentation process, overcoming the limitations of SAM in obtaining suitable prompts for nuclei segmentation tasks. Through extensive experiments, SAC demonstrates superior performance in nuclei segmentation tasks, proving its effectiveness as a tool for pathologists and researchers. The SAC framework includes a LoRA implementation within the SAM model to enhance the fine-tuning process, which outperforms existing fine-tuning/adapter-based SOTA methods. The auto-prompt generator is designed to automatically generate a large number of high-quality prompts to guide the segmentation process. This addresses the limitation of SAM's difficulty in obtaining suitable prompts for nuclei segmentation tasks. SAC's method is fully automated and easily adaptable to various nuclei segmentation tasks, offering a simplified yet effective tool for pathologists and researchers. SAC's auto-prompt generator is an auxiliary neural network that automates the process of generating prompts. It uses an auxiliary neural network to generate masks from input images, which are then used to create positive and negative prompts. These prompts are fed into the SAM prompt encoder to obtain prompt embeddings, which are then used in the SAM mask decoder to produce the final segmentation results. The auto-prompt generator also includes prompt discrimination capabilities, which are crucial for cell nuclei segmentation as SAM's ability to distinguish between positive and negative prompts significantly impacts segmentation accuracy. SAC's method is evaluated on two datasets: MoNuSeg and the 2018 Data Science Bowl (DSB). The results show that SAC outperforms existing SOTA methods in nuclei segmentation tasks, demonstrating its effectiveness and superiority. The method's performance is measured using F1 score, Dice coefficient, and Intersection over Union (IoU). SAC's auto-prompt generator and LoRA implementation significantly enhance the model's performance in nuclei segmentation tasks, making it a valuable tool for medical image analysis. The framework's ability to automatically generate prompts and adapt to various segmentation tasks highlights its generalizability and flexibility.
Reach us at info@study.space