**ThinkGrasp: A Vision-Language System for Strategic Part Grasping in Clutter** **Authors:** Yaoyao Qian, Xupeng Zhu, Ondrej Biza, Shuo Jiang, Linfeng Zhao, Haojie Huang, Yu Qi, Robert Platt **Institution:** Northeastern University, Boston, MA, USA; Boston Dynamics AI Institute **Abstract:** Robotic grasping in cluttered environments remains a significant challenge due to occlusions and complex object arrangements. ThinkGrasp is a plug-and-play vision-language grasping system that leverages GPT-4o's advanced contextual reasoning to handle heavy clutter environments. It can effectively identify and generate grasp poses for target objects, even when they are heavily obstructed or nearly invisible, by using goal-oriented language to guide the removal of obstructing objects. This approach progressively uncovers the target object and ultimately grasps it with a few steps and a high success rate. In both simulated and real experiments, ThinkGrasp achieved a high success rate and significantly outperformed state-of-the-art methods in heavily cluttered environments or with diverse unseen objects, demonstrating strong generalization capabilities. **Keywords:** Robotic Grasping, Vision-Language Models, Language Conditioned Grasping **Introduction:** The field of robotic grasping has seen significant advancements, but grasping in highly cluttered environments remains a major challenge. ThinkGrasp combines the strengths of large-scale pre-trained vision-language models with an occlusion handling system. It leverages GPT-4o's advanced reasoning capabilities to gain a visual understanding of environmental and object properties, enhancing success rates and ensuring safe grasp poses by strategically eliminating obstructing objects. **Contributions:** - Developed a plug-and-play system for occlusion handling that efficiently utilizes visual and language information. - Implemented a robust error-handling framework using LangSAM and VLPart for segmentation. - Achieved state-of-the-art performance in simulation and real-world experiments, outperforming prior methods in cluttered scenes and with unseen objects. **Method:** ThinkGrasp uses an iterative pipeline that includes: - **Problem Definition:** Addressing challenges in occlusions, ambiguity in natural language instructions, and dynamic environments. - **System Pipeline:** Utilizing GPT-4o for "imagine segmentation," LangSAM or VLPart for segmentation, and a closed-loop process for robustness. - **GPT-4o’s Role:** Seeks the most relevant object based on the instruction, handles occlusions, and selects optimal grasp regions using a 3×3 grid strategy. - **Grasp Pose Generation:** Evaluates candidate grasp poses based on proximity to the preferred location and grasp quality scores. **Experiments:** - **Simulation:** Compared against state-of-the-art methods, achieving a 98.0% success rate and fewer steps in cluttered scenes. - **Real-World:** Extended to real-world environments, demonstrating high success rates**ThinkGrasp: A Vision-Language System for Strategic Part Grasping in Clutter** **Authors:** Yaoyao Qian, Xupeng Zhu, Ondrej Biza, Shuo Jiang, Linfeng Zhao, Haojie Huang, Yu Qi, Robert Platt **Institution:** Northeastern University, Boston, MA, USA; Boston Dynamics AI Institute **Abstract:** Robotic grasping in cluttered environments remains a significant challenge due to occlusions and complex object arrangements. ThinkGrasp is a plug-and-play vision-language grasping system that leverages GPT-4o's advanced contextual reasoning to handle heavy clutter environments. It can effectively identify and generate grasp poses for target objects, even when they are heavily obstructed or nearly invisible, by using goal-oriented language to guide the removal of obstructing objects. This approach progressively uncovers the target object and ultimately grasps it with a few steps and a high success rate. In both simulated and real experiments, ThinkGrasp achieved a high success rate and significantly outperformed state-of-the-art methods in heavily cluttered environments or with diverse unseen objects, demonstrating strong generalization capabilities. **Keywords:** Robotic Grasping, Vision-Language Models, Language Conditioned Grasping **Introduction:** The field of robotic grasping has seen significant advancements, but grasping in highly cluttered environments remains a major challenge. ThinkGrasp combines the strengths of large-scale pre-trained vision-language models with an occlusion handling system. It leverages GPT-4o's advanced reasoning capabilities to gain a visual understanding of environmental and object properties, enhancing success rates and ensuring safe grasp poses by strategically eliminating obstructing objects. **Contributions:** - Developed a plug-and-play system for occlusion handling that efficiently utilizes visual and language information. - Implemented a robust error-handling framework using LangSAM and VLPart for segmentation. - Achieved state-of-the-art performance in simulation and real-world experiments, outperforming prior methods in cluttered scenes and with unseen objects. **Method:** ThinkGrasp uses an iterative pipeline that includes: - **Problem Definition:** Addressing challenges in occlusions, ambiguity in natural language instructions, and dynamic environments. - **System Pipeline:** Utilizing GPT-4o for "imagine segmentation," LangSAM or VLPart for segmentation, and a closed-loop process for robustness. - **GPT-4o’s Role:** Seeks the most relevant object based on the instruction, handles occlusions, and selects optimal grasp regions using a 3×3 grid strategy. - **Grasp Pose Generation:** Evaluates candidate grasp poses based on proximity to the preferred location and grasp quality scores. **Experiments:** - **Simulation:** Compared against state-of-the-art methods, achieving a 98.0% success rate and fewer steps in cluttered scenes. - **Real-World:** Extended to real-world environments, demonstrating high success rates