**Manipulate-Anything: Automating Real-World Robots using Vision-Language Models** The paper introduces MANIPULATE-ANYTHING, a scalable and environment-agnostic method for generating zero-shot demonstrations for robotic manipulation tasks. Unlike previous methods, MANIPULATE-ANYTHING can operate in real-world environments without privileged state information, hand-designed skills, or limitations on the number of object instances. The method leverages vision-language models (VLMs) to decompose tasks into sub-tasks, generate actions, and verify their success. It uses multi-viewpoint reasoning to improve performance and includes error recovery mechanisms to handle failures. **Key Contributions:** 1. **Zero-shot Performance:** MANIPULATE-ANYTHING successfully generates trajectories for 7 real-world and 14 simulation tasks, outperforming existing methods like VoxPoser. 2. **Behavior Cloning:** The generated demonstrations enable training more robust behavior cloning policies, outperforming human demonstrations and other baselines. 3. **Scalability:** The method can generate large quantities of high-quality data for robotics, addressing the lack of diverse and sufficient data in robotics. **Methods:** - **Task Plan Generation:** Uses VLMs to decompose tasks into sub-tasks and generate action sequences. - **Multi-viewpoint VLM Selection:** Chooses optimal viewpoints for action generation and sub-task verification. - **Action Generation:** Generates low-level actions for agent-centric and object-centric tasks. - **Sub-task Verification:** Verifies the success of each sub-task using a VLM-based verifier. **Experiments:** - **Zero-shot Performance:** Outperforms baselines in 10 out of 14 simulation tasks. - **Behavior Cloning:** Trained policies using MANIPULATE-ANYTHING data perform similarly to those trained on human demonstrations. - **Real-world Experiments:** Achieves a success rate of over 25% for 7 real-world tasks, outperforming CAP by 38%. **Discussion:** - **Limitations:** Relies on large language models, struggles with dynamic and non-prehensile tasks, and requires manual prompt engineering. - **Future Work:** Potential improvements include specialized VLMs and advanced alignment techniques. **Conclusion:** MANIPULATE-ANYTHING is a promising method for generating high-quality, diverse, and scalable data for robotic manipulation tasks, enabling better performance in zero-shot settings.**Manipulate-Anything: Automating Real-World Robots using Vision-Language Models** The paper introduces MANIPULATE-ANYTHING, a scalable and environment-agnostic method for generating zero-shot demonstrations for robotic manipulation tasks. Unlike previous methods, MANIPULATE-ANYTHING can operate in real-world environments without privileged state information, hand-designed skills, or limitations on the number of object instances. The method leverages vision-language models (VLMs) to decompose tasks into sub-tasks, generate actions, and verify their success. It uses multi-viewpoint reasoning to improve performance and includes error recovery mechanisms to handle failures. **Key Contributions:** 1. **Zero-shot Performance:** MANIPULATE-ANYTHING successfully generates trajectories for 7 real-world and 14 simulation tasks, outperforming existing methods like VoxPoser. 2. **Behavior Cloning:** The generated demonstrations enable training more robust behavior cloning policies, outperforming human demonstrations and other baselines. 3. **Scalability:** The method can generate large quantities of high-quality data for robotics, addressing the lack of diverse and sufficient data in robotics. **Methods:** - **Task Plan Generation:** Uses VLMs to decompose tasks into sub-tasks and generate action sequences. - **Multi-viewpoint VLM Selection:** Chooses optimal viewpoints for action generation and sub-task verification. - **Action Generation:** Generates low-level actions for agent-centric and object-centric tasks. - **Sub-task Verification:** Verifies the success of each sub-task using a VLM-based verifier. **Experiments:** - **Zero-shot Performance:** Outperforms baselines in 10 out of 14 simulation tasks. - **Behavior Cloning:** Trained policies using MANIPULATE-ANYTHING data perform similarly to those trained on human demonstrations. - **Real-world Experiments:** Achieves a success rate of over 25% for 7 real-world tasks, outperforming CAP by 38%. **Discussion:** - **Limitations:** Relies on large language models, struggles with dynamic and non-prehensile tasks, and requires manual prompt engineering. - **Future Work:** Potential improvements include specialized VLMs and advanced alignment techniques. **Conclusion:** MANIPULATE-ANYTHING is a promising method for generating high-quality, diverse, and scalable data for robotic manipulation tasks, enabling better performance in zero-shot settings.