Manipulate-Anything: Automating Real-World Robots using Vision-Language Models This paper introduces MANIPULATE-ANYTHING, a scalable automated method for generating real-world robotic manipulation data. Unlike prior methods, MANIPULATE-ANYTHING operates in real-world environments without privileged state information or hand-designed skills, and can manipulate any static object. The method uses vision-language models (VLMs) to decompose tasks into sub-tasks, generate code for new skills or task-specific grasp poses, and verify the success of each sub-task. It can guide a robot to accomplish a diverse set of unseen tasks, manipulating diverse objects. The generated data enables training behavior cloning policies that outperform training with human demonstrations or data generated by VoxPoser, Scaling-up, and Code-As-Policies. The method is evaluated using two setups. First, MANIPULATE-ANYTHING successfully generates trajectories for all 7 real-world and 14 simulation tasks, significantly outperforming existing methods like VoxPoser. Second, MANIPULATE-ANYTHING's demonstrations can train more robust behavior cloning policies than training with human demonstrations or data generated by VoxPoser, Scaling-up, and Code-As-Policies. The method demonstrates the broad possibility of large-scale deployment of robots across unstructured real-world environments and highlights its utility as a training data generator, aiding in the crucial goal of scaling up robot demonstration data. The paper also discusses the limitations of MANIPULATE-ANYTHING, including its dependence on large language models (LLMs), struggles with dynamic manipulation tasks and non-prehensile tasks, and the potential for compounding errors when generating zero-shot trajectories. The authors suggest that emerging specialized VLMs may help address these issues. Additionally, manual prompt engineering for in-context learning is still required, but recent advancements in alignment and prompting techniques offer potential solutions to reduce the effort involved in prompt engineering. In conclusion, MANIPULATE-ANYTHING is a scalable environment-agnostic approach for generating zero-shot demonstration for robotic tasks without the use of privileged environment information. It uses LLMs to do high level planning and scene understanding and is capable of error recovery. This enables high quality data generation for behavior cloning that can achieve better performance than using human data.Manipulate-Anything: Automating Real-World Robots using Vision-Language Models This paper introduces MANIPULATE-ANYTHING, a scalable automated method for generating real-world robotic manipulation data. Unlike prior methods, MANIPULATE-ANYTHING operates in real-world environments without privileged state information or hand-designed skills, and can manipulate any static object. The method uses vision-language models (VLMs) to decompose tasks into sub-tasks, generate code for new skills or task-specific grasp poses, and verify the success of each sub-task. It can guide a robot to accomplish a diverse set of unseen tasks, manipulating diverse objects. The generated data enables training behavior cloning policies that outperform training with human demonstrations or data generated by VoxPoser, Scaling-up, and Code-As-Policies. The method is evaluated using two setups. First, MANIPULATE-ANYTHING successfully generates trajectories for all 7 real-world and 14 simulation tasks, significantly outperforming existing methods like VoxPoser. Second, MANIPULATE-ANYTHING's demonstrations can train more robust behavior cloning policies than training with human demonstrations or data generated by VoxPoser, Scaling-up, and Code-As-Policies. The method demonstrates the broad possibility of large-scale deployment of robots across unstructured real-world environments and highlights its utility as a training data generator, aiding in the crucial goal of scaling up robot demonstration data. The paper also discusses the limitations of MANIPULATE-ANYTHING, including its dependence on large language models (LLMs), struggles with dynamic manipulation tasks and non-prehensile tasks, and the potential for compounding errors when generating zero-shot trajectories. The authors suggest that emerging specialized VLMs may help address these issues. Additionally, manual prompt engineering for in-context learning is still required, but recent advancements in alignment and prompting techniques offer potential solutions to reduce the effort involved in prompt engineering. In conclusion, MANIPULATE-ANYTHING is a scalable environment-agnostic approach for generating zero-shot demonstration for robotic tasks without the use of privileged environment information. It uses LLMs to do high level planning and scene understanding and is capable of error recovery. This enables high quality data generation for behavior cloning that can achieve better performance than using human data.