Grounded SAM is an innovative framework that integrates the Grounding DINO object detector with the Segment Anything Model (SAM) to enable open-set detection and segmentation based on arbitrary text inputs. This integration allows for a wide range of vision tasks, including automatic image annotation, controllable image editing, and promptable 3D human motion analysis. Grounded SAM demonstrates superior performance on open-vocabulary benchmarks, achieving 48.7 mean AP on the SegInW zero-shot benchmark with the combination of Grounding DINO-Base and SAM-Huge models. The framework addresses the challenges in open-world visual perception by decoupling complex tasks into manageable sub-tasks, leveraging the strengths of various expert models. It can be extended with additional models such as BLIP, Recognize Anything (RAM), Stable Diffusion, and OSX to achieve more sophisticated functionalities. Grounded SAM's effectiveness is validated through experiments on the Segmentation in the Wild (SGinW) benchmark, showing significant improvements over previous unified open-set segmentation models. The paper also discusses the potential future applications of Grounded SAM, including establishing a closed-loop system for annotation and model training, integrating with large language models, and generating new datasets. The contributions of the project are acknowledged, highlighting the collaborative efforts of multiple researchers and contributors.Grounded SAM is an innovative framework that integrates the Grounding DINO object detector with the Segment Anything Model (SAM) to enable open-set detection and segmentation based on arbitrary text inputs. This integration allows for a wide range of vision tasks, including automatic image annotation, controllable image editing, and promptable 3D human motion analysis. Grounded SAM demonstrates superior performance on open-vocabulary benchmarks, achieving 48.7 mean AP on the SegInW zero-shot benchmark with the combination of Grounding DINO-Base and SAM-Huge models. The framework addresses the challenges in open-world visual perception by decoupling complex tasks into manageable sub-tasks, leveraging the strengths of various expert models. It can be extended with additional models such as BLIP, Recognize Anything (RAM), Stable Diffusion, and OSX to achieve more sophisticated functionalities. Grounded SAM's effectiveness is validated through experiments on the Segmentation in the Wild (SGinW) benchmark, showing significant improvements over previous unified open-set segmentation models. The paper also discusses the potential future applications of Grounded SAM, including establishing a closed-loop system for annotation and model training, integrating with large language models, and generating new datasets. The contributions of the project are acknowledged, highlighting the collaborative efforts of multiple researchers and contributors.