This paper introduces a new fully annotated photorealistic synthetic video dataset called Virtual KITTI, built using modern computer graphics technology and a novel real-to-virtual cloning method. The dataset contains 35 photo-realistic synthetic videos, with 5 cloned from the original real-world KITTI tracking benchmark, and 7 variations each, totaling approximately 17,000 high-resolution frames. Each frame is automatically labeled with accurate ground truth for object detection, tracking, depth, optical flow, and scene and instance segmentation. The authors propose an efficient real-to-virtual world cloning method and validate their approach by building and publicly releasing the Virtual KITTI dataset. They provide quantitative experimental evidence suggesting that (i) modern deep learning algorithms pre-trained on real data behave similarly in real and virtual worlds, and (ii) pre-training on virtual data improves performance. The small gap between real and virtual worlds enables measuring the impact of various weather and imaging conditions on recognition performance. The paper evaluates the transferability of experimental observations across real and virtual worlds, the benefits of virtual pre-training, and the impact of various weather and imaging conditions on recognition performance. The experiments show that virtual pre-training followed by real-world fine-tuning outperforms training only on real-world data. The results also indicate that varied weather conditions, such as fog, lighting conditions, and camera angles, can significantly deteriorate the performance of normally high-performing models trained on large real-world datasets. The authors also assess the usefulness of virtual worlds as proxies for multi-object tracking. They propose a practical definition of transferability of experimental observations across real and virtual worlds, using real-world pre-trained deep models, hyper-parameter calibration via Bayesian optimization, and the analysis of task-specific performance metrics. The results suggest that recent progress in computer graphics technology allows one to easily build virtual worlds that are indeed effective proxies of the real world from a computer vision perspective. The paper concludes that virtual worlds can be used to measure the potential impact of various factors on recognition performance, and that virtual pre-training can improve the performance of deep learning models. The authors plan to expand Virtual KITTI by adding more worlds and including pedestrians, and to explore and evaluate domain adaptation methods and larger scale virtual pre-training or data augmentation to build more robust models for a variety of video understanding tasks.This paper introduces a new fully annotated photorealistic synthetic video dataset called Virtual KITTI, built using modern computer graphics technology and a novel real-to-virtual cloning method. The dataset contains 35 photo-realistic synthetic videos, with 5 cloned from the original real-world KITTI tracking benchmark, and 7 variations each, totaling approximately 17,000 high-resolution frames. Each frame is automatically labeled with accurate ground truth for object detection, tracking, depth, optical flow, and scene and instance segmentation. The authors propose an efficient real-to-virtual world cloning method and validate their approach by building and publicly releasing the Virtual KITTI dataset. They provide quantitative experimental evidence suggesting that (i) modern deep learning algorithms pre-trained on real data behave similarly in real and virtual worlds, and (ii) pre-training on virtual data improves performance. The small gap between real and virtual worlds enables measuring the impact of various weather and imaging conditions on recognition performance. The paper evaluates the transferability of experimental observations across real and virtual worlds, the benefits of virtual pre-training, and the impact of various weather and imaging conditions on recognition performance. The experiments show that virtual pre-training followed by real-world fine-tuning outperforms training only on real-world data. The results also indicate that varied weather conditions, such as fog, lighting conditions, and camera angles, can significantly deteriorate the performance of normally high-performing models trained on large real-world datasets. The authors also assess the usefulness of virtual worlds as proxies for multi-object tracking. They propose a practical definition of transferability of experimental observations across real and virtual worlds, using real-world pre-trained deep models, hyper-parameter calibration via Bayesian optimization, and the analysis of task-specific performance metrics. The results suggest that recent progress in computer graphics technology allows one to easily build virtual worlds that are indeed effective proxies of the real world from a computer vision perspective. The paper concludes that virtual worlds can be used to measure the potential impact of various factors on recognition performance, and that virtual pre-training can improve the performance of deep learning models. The authors plan to expand Virtual KITTI by adding more worlds and including pedestrians, and to explore and evaluate domain adaptation methods and larger scale virtual pre-training or data augmentation to build more robust models for a variety of video understanding tasks.