24 Apr 2024 | Qinxi Yu, Masoud Moghani, Karthik Dharmanajan, Vincent Schorp, William Chung-Ho Panitch, Jingzhou Liu, Kush Hari, Huang Huang, Mayank Mittal, Ken Goldberg, Animesh Garg
ORBIT-Surgical is an open-source physics-based simulation framework for surgical robotics, developed using NVIDIA Isaac Sim. It provides 14 benchmark surgical tasks for the da Vinci Research Kit (dVRK) and Smart Tissue Autonomous Robot (STAR), representing common subtasks in surgical training. The framework supports GPU-accelerated physics, realistic rendering, and a modular programming interface, enabling the study of robot learning to augment human surgical skills. ORBIT-Surgical facilitates the generation of realistic synthetic data for active perception tasks and demonstrates sim-to-real transfer of learned policies onto a physical dVRK robot.
The framework includes a variety of surgical tasks, such as reaching, lifting, and transferring surgical instruments, as well as tasks involving deformable objects. It supports both reinforcement learning (RL) and imitation learning (IL) algorithms, as well as teleoperation and synthetic data generation. ORBIT-Surgical is designed to be modular, allowing researchers to quickly evaluate new algorithms on detailed surgical scenes.
ORBIT-Surgical leverages GPU-based physics engines and signed-distance field (SDF) collision checking for realistic simulations of surgical environments. It supports a wide range of robot learning workflows, including RL and IL, teleoperation, and synthetic data generation. The framework also includes a rich suite of rigid and deformable object models, as well as a standardized programming interface.
The paper presents experimental results demonstrating the effectiveness of ORBIT-Surgical in simulating surgical tasks and transferring learned policies to a physical dVRK robot. It shows that ORBIT-Surgical can train RL policies for surgical tasks in a fraction of the time required by other simulators, and that policies trained in simulation can be successfully transferred to real-world robots. The framework also supports the generation of synthetic data for training surgical tool segmentation models, which improves performance by over 2× when combined with real data.
ORBIT-Surgical is an open-source framework that provides a realistic simulation environment for surgical robotics, enabling the study of robot learning and the development of policies for surgical tasks. It supports a wide range of surgical tasks and is designed to be modular, allowing researchers to quickly evaluate new algorithms on detailed surgical scenes. The framework is built on top of NVIDIA Isaac Sim and provides a unified simulation interface for robot learning. It supports a variety of robot learning workflows, including RL and IL, teleoperation, and synthetic data generation. The framework is designed to be efficient, with the ability to simulate up to 8000 environments in parallel on a single NVIDIA RTX 3090 GPU.ORBIT-Surgical is an open-source physics-based simulation framework for surgical robotics, developed using NVIDIA Isaac Sim. It provides 14 benchmark surgical tasks for the da Vinci Research Kit (dVRK) and Smart Tissue Autonomous Robot (STAR), representing common subtasks in surgical training. The framework supports GPU-accelerated physics, realistic rendering, and a modular programming interface, enabling the study of robot learning to augment human surgical skills. ORBIT-Surgical facilitates the generation of realistic synthetic data for active perception tasks and demonstrates sim-to-real transfer of learned policies onto a physical dVRK robot.
The framework includes a variety of surgical tasks, such as reaching, lifting, and transferring surgical instruments, as well as tasks involving deformable objects. It supports both reinforcement learning (RL) and imitation learning (IL) algorithms, as well as teleoperation and synthetic data generation. ORBIT-Surgical is designed to be modular, allowing researchers to quickly evaluate new algorithms on detailed surgical scenes.
ORBIT-Surgical leverages GPU-based physics engines and signed-distance field (SDF) collision checking for realistic simulations of surgical environments. It supports a wide range of robot learning workflows, including RL and IL, teleoperation, and synthetic data generation. The framework also includes a rich suite of rigid and deformable object models, as well as a standardized programming interface.
The paper presents experimental results demonstrating the effectiveness of ORBIT-Surgical in simulating surgical tasks and transferring learned policies to a physical dVRK robot. It shows that ORBIT-Surgical can train RL policies for surgical tasks in a fraction of the time required by other simulators, and that policies trained in simulation can be successfully transferred to real-world robots. The framework also supports the generation of synthetic data for training surgical tool segmentation models, which improves performance by over 2× when combined with real data.
ORBIT-Surgical is an open-source framework that provides a realistic simulation environment for surgical robotics, enabling the study of robot learning and the development of policies for surgical tasks. It supports a wide range of surgical tasks and is designed to be modular, allowing researchers to quickly evaluate new algorithms on detailed surgical scenes. The framework is built on top of NVIDIA Isaac Sim and provides a unified simulation interface for robot learning. It supports a variety of robot learning workflows, including RL and IL, teleoperation, and synthetic data generation. The framework is designed to be efficient, with the ability to simulate up to 8000 environments in parallel on a single NVIDIA RTX 3090 GPU.