RialTo is a system that improves the robustness of real-world imitation learning policies by using reinforcement learning (RL) in simulation environments constructed from small amounts of real-world data. The system enables a real-to-sim-to-real pipeline by providing an easy-to-use interface for quickly scanning and constructing digital twins of real-world environments. It also introduces a novel "inverse distillation" procedure to transfer real-world demonstrations into simulated environments for efficient fine-tuning with minimal human intervention. RialTo is evaluated across a variety of robotic manipulation tasks in the real world, such as stacking dishes on a rack, placing books on a shelf, and six other tasks. RialTo increases policy robustness by over 67% without requiring extensive human data collection. The system's key insight is to train RL controllers on quickly constructed simulation scenes, leveraging video from the target deployment domain to obtain scenes with accurate geometry and articulation that reflect the appearance and kinematics of the real world. These "in-domain" simulation environments can serve as a sandbox to safely and quickly learn robust policies across various disturbances and distractors without requiring expensive exploration in the real world. RialTo's pipeline simultaneously improves the effectiveness of both reinforcement and imitation learning. RL in simulation helps make imitation learning policies deployment-ready without requiring prohibitive amounts of unsafe, interactive data collection in the real world. At the same time, bootstrapping from real-world demonstration data via inverse distillation makes the exploration problem tractable for RL fine-tuning in simulation. This minimizes the amount of task-specific engineering required by algorithm designers. RialTo's contributions include a simple policy learning pipeline that synthesizes controllers to perform diverse manipulation tasks in the real world with reduced human effort, a novel algorithm for transferring demonstrations from the real world to the reconstructed simulation to bootstrap efficient reinforcement learning, and an intuitive graphical interface for quickly scanning and constructing digital twins of real-world scenes. The system is evaluated across eight diverse tasks, showing a 67% improvement in average success rate across scenarios with varying object poses, visual distractors, and physical perturbations.RialTo is a system that improves the robustness of real-world imitation learning policies by using reinforcement learning (RL) in simulation environments constructed from small amounts of real-world data. The system enables a real-to-sim-to-real pipeline by providing an easy-to-use interface for quickly scanning and constructing digital twins of real-world environments. It also introduces a novel "inverse distillation" procedure to transfer real-world demonstrations into simulated environments for efficient fine-tuning with minimal human intervention. RialTo is evaluated across a variety of robotic manipulation tasks in the real world, such as stacking dishes on a rack, placing books on a shelf, and six other tasks. RialTo increases policy robustness by over 67% without requiring extensive human data collection. The system's key insight is to train RL controllers on quickly constructed simulation scenes, leveraging video from the target deployment domain to obtain scenes with accurate geometry and articulation that reflect the appearance and kinematics of the real world. These "in-domain" simulation environments can serve as a sandbox to safely and quickly learn robust policies across various disturbances and distractors without requiring expensive exploration in the real world. RialTo's pipeline simultaneously improves the effectiveness of both reinforcement and imitation learning. RL in simulation helps make imitation learning policies deployment-ready without requiring prohibitive amounts of unsafe, interactive data collection in the real world. At the same time, bootstrapping from real-world demonstration data via inverse distillation makes the exploration problem tractable for RL fine-tuning in simulation. This minimizes the amount of task-specific engineering required by algorithm designers. RialTo's contributions include a simple policy learning pipeline that synthesizes controllers to perform diverse manipulation tasks in the real world with reduced human effort, a novel algorithm for transferring demonstrations from the real world to the reconstructed simulation to bootstrap efficient reinforcement learning, and an intuitive graphical interface for quickly scanning and constructing digital twins of real-world scenes. The system is evaluated across eight diverse tasks, showing a 67% improvement in average success rate across scenarios with varying object poses, visual distractors, and physical perturbations.