The paper introduces PA-SAM, a novel prompt-driven adapter for the Segment Anything Model (SAM), aimed at enhancing the quality of segmentation masks. PA-SAM specifically addresses the limitations of SAM in producing high-quality masks, particularly in real-world scenarios. By training a prompt adapter to extract detailed information from images and optimize the mask decoder feature at both sparse and dense prompt levels, PA-SAM improves SAM's segmentation performance. Experimental results show that PA-SAM outperforms other SAM-based methods in high-quality, zero-shot, and open-set segmentation tasks, achieving significant improvements in metrics such as mIoU and BmIoU. The method is evaluated on various datasets, including HQSeg-44K, COCO, and SegInW, demonstrating its effectiveness in handling challenging scenarios and improving segmentation accuracy. The source code and models are available at <https://github.com/xzz2/pa-sam>.The paper introduces PA-SAM, a novel prompt-driven adapter for the Segment Anything Model (SAM), aimed at enhancing the quality of segmentation masks. PA-SAM specifically addresses the limitations of SAM in producing high-quality masks, particularly in real-world scenarios. By training a prompt adapter to extract detailed information from images and optimize the mask decoder feature at both sparse and dense prompt levels, PA-SAM improves SAM's segmentation performance. Experimental results show that PA-SAM outperforms other SAM-based methods in high-quality, zero-shot, and open-set segmentation tasks, achieving significant improvements in metrics such as mIoU and BmIoU. The method is evaluated on various datasets, including HQSeg-44K, COCO, and SegInW, demonstrating its effectiveness in handling challenging scenarios and improving segmentation accuracy. The source code and models are available at <https://github.com/xzz2/pa-sam>.