July 2024 | Jianlan Luo, Charles Xu, Fangchen Liu, Liam Tan, Zipeng Lin, Jeffrey Wu, Pieter Abbeel and Sergey Levine
This paper introduces the Functional Manipulation Benchmark (FMB), a real-world benchmark for studying robotic learning in the context of functional manipulation. The FMB is designed to balance complexity and accessibility, offering tasks that are narrow enough to be effectively studied with manageable datasets but diverse enough to challenge generalization. The benchmark includes 66 3D-printed objects, procedurally generated to allow for controlled generalization studies. Tasks involve fundamental manipulation skills such as grasping, repositioning, and assembly, requiring robots to perform complex, multi-stage tasks by composing individual skills. The FMB includes an imitation learning framework with policies trained to solve these tasks, enabling researchers to use the benchmark as a versatile toolkit for evaluating various aspects of robotic learning. The benchmark is designed to be easily replicable, with all necessary hardware and software components provided. The dataset includes 22,500 human demonstrations of grasping, repositioning, and assembly skills, along with RGB, depth, and force/torque data. The FMB allows researchers to evaluate methods for acquiring individual skills and combining them to solve complex tasks. The benchmark includes both single-object and multi-object manipulation tasks, with the latter involving interlocking objects and more complex assembly. The FMB is modular, allowing researchers to focus on specific aspects of the tasks. The paper also discusses related work, the design of the benchmark, and the evaluation protocols for different tasks. The results show that the FMB provides a valuable resource for studying robotic manipulation, with policies trained on the benchmark demonstrating varying levels of success across different tasks and input modalities. The benchmark is intended to facilitate progress in robotic learning by providing a standardized toolkit for evaluating generalization and physical complexity in manipulation tasks.This paper introduces the Functional Manipulation Benchmark (FMB), a real-world benchmark for studying robotic learning in the context of functional manipulation. The FMB is designed to balance complexity and accessibility, offering tasks that are narrow enough to be effectively studied with manageable datasets but diverse enough to challenge generalization. The benchmark includes 66 3D-printed objects, procedurally generated to allow for controlled generalization studies. Tasks involve fundamental manipulation skills such as grasping, repositioning, and assembly, requiring robots to perform complex, multi-stage tasks by composing individual skills. The FMB includes an imitation learning framework with policies trained to solve these tasks, enabling researchers to use the benchmark as a versatile toolkit for evaluating various aspects of robotic learning. The benchmark is designed to be easily replicable, with all necessary hardware and software components provided. The dataset includes 22,500 human demonstrations of grasping, repositioning, and assembly skills, along with RGB, depth, and force/torque data. The FMB allows researchers to evaluate methods for acquiring individual skills and combining them to solve complex tasks. The benchmark includes both single-object and multi-object manipulation tasks, with the latter involving interlocking objects and more complex assembly. The FMB is modular, allowing researchers to focus on specific aspects of the tasks. The paper also discusses related work, the design of the benchmark, and the evaluation protocols for different tasks. The results show that the FMB provides a valuable resource for studying robotic manipulation, with policies trained on the benchmark demonstrating varying levels of success across different tasks and input modalities. The benchmark is intended to facilitate progress in robotic learning by providing a standardized toolkit for evaluating generalization and physical complexity in manipulation tasks.