**RoadRunner - Learning Traversability Estimation for Autonomous Off-road Driving** **Authors:** - Jonas Frey - Shehryar Khattak - Manthan Patel - Deegan Atha - Julian Nubert - Curtis Padgett - Marco Hutter - Patrick Spieler **Abstract:** This paper presents RoadRunner, a novel framework for predicting terrain traversability and elevation maps directly from camera and LiDAR sensor inputs. RoadRunner enables reliable autonomous navigation by fusing sensory information, handling uncertainty, and generating contextually informed predictions about terrain geometry and traversability at low latency. Unlike existing methods that rely on handcrafted semantic classes and heuristics, RoadRunner is trained end-to-end in a self-supervised manner. The network architecture builds upon popular sensor fusion networks from autonomous driving, combining LiDAR and camera information into a common Bird's Eye View (BEV) perspective. Training is achieved using an existing traversability estimation stack (X-Racer) to generate training data in hindsight from real-world off-road driving datasets. RoadRunner improves system latency by a factor of ~4 (from 500 ms to 140 ms) while enhancing accuracy in traversability cost and elevation map predictions. The effectiveness of RoadRunner is demonstrated through safe and reliable off-road navigation at high speeds in multiple real-world driving scenarios. **Key Contributions:** - A novel RoadRunner network architecture for predicting traversability costs and elevation maps from multi-LiDAR and multi-camera data at low latency. - A general framework for generating pseudo-ground truth elevation maps and traversability costs using temporal data aggregation in hindsight for self-supervised training. - An overview of the NASA Jet Propulsion Laboratory's X-Racer off-road autonomy research stack. - An extensive evaluation of RoadRunner on real-world field test data, comparing it to existing network architectures and conducting ablation studies. **Related Work:** The paper reviews existing methods for traversability estimation in off-road driving, focusing on semantic segmentation, self-supervision, and other approaches. It highlights the limitations of current methods, such as reliance on expensive annotated data and the need for precise GPS guidance, and discusses the benefits of RoadRunner's self-supervised approach and low latency. **Method:** The RoadRunner network is designed to predict vehicle-centric elevation and traversability cost maps using multi-camera and multi-LiDAR inputs. The network architecture combines features from camera images and LiDAR point clouds, leveraging pre-trained models like EfficientNet-B0, PointPillars, and Lift Splat Shoot. The network is trained using a Weighted Mean Squared Error (WMSE) loss function, with weights adjusted based on the frequency of traversability and elevation values. **Experiments:** The dataset for training RoadRunner consists of 16.5 km of off-road driving data collected in various scenarios, including open fields, dirt roads, and forested environments.**RoadRunner - Learning Traversability Estimation for Autonomous Off-road Driving** **Authors:** - Jonas Frey - Shehryar Khattak - Manthan Patel - Deegan Atha - Julian Nubert - Curtis Padgett - Marco Hutter - Patrick Spieler **Abstract:** This paper presents RoadRunner, a novel framework for predicting terrain traversability and elevation maps directly from camera and LiDAR sensor inputs. RoadRunner enables reliable autonomous navigation by fusing sensory information, handling uncertainty, and generating contextually informed predictions about terrain geometry and traversability at low latency. Unlike existing methods that rely on handcrafted semantic classes and heuristics, RoadRunner is trained end-to-end in a self-supervised manner. The network architecture builds upon popular sensor fusion networks from autonomous driving, combining LiDAR and camera information into a common Bird's Eye View (BEV) perspective. Training is achieved using an existing traversability estimation stack (X-Racer) to generate training data in hindsight from real-world off-road driving datasets. RoadRunner improves system latency by a factor of ~4 (from 500 ms to 140 ms) while enhancing accuracy in traversability cost and elevation map predictions. The effectiveness of RoadRunner is demonstrated through safe and reliable off-road navigation at high speeds in multiple real-world driving scenarios. **Key Contributions:** - A novel RoadRunner network architecture for predicting traversability costs and elevation maps from multi-LiDAR and multi-camera data at low latency. - A general framework for generating pseudo-ground truth elevation maps and traversability costs using temporal data aggregation in hindsight for self-supervised training. - An overview of the NASA Jet Propulsion Laboratory's X-Racer off-road autonomy research stack. - An extensive evaluation of RoadRunner on real-world field test data, comparing it to existing network architectures and conducting ablation studies. **Related Work:** The paper reviews existing methods for traversability estimation in off-road driving, focusing on semantic segmentation, self-supervision, and other approaches. It highlights the limitations of current methods, such as reliance on expensive annotated data and the need for precise GPS guidance, and discusses the benefits of RoadRunner's self-supervised approach and low latency. **Method:** The RoadRunner network is designed to predict vehicle-centric elevation and traversability cost maps using multi-camera and multi-LiDAR inputs. The network architecture combines features from camera images and LiDAR point clouds, leveraging pre-trained models like EfficientNet-B0, PointPillars, and Lift Splat Shoot. The network is trained using a Weighted Mean Squared Error (WMSE) loss function, with weights adjusted based on the frequency of traversability and elevation values. **Experiments:** The dataset for training RoadRunner consists of 16.5 km of off-road driving data collected in various scenarios, including open fields, dirt roads, and forested environments.