NeuroNCAP is a photorealistic closed-loop simulator for testing autonomous driving (AD) software systems, designed to evaluate safety-critical scenarios. The simulator uses NeRF (Neural Radiance Fields) to generate realistic sensor data from real-world driving logs, enabling the creation of new, unseen scenarios. It allows for reconfiguration and rendering of various safety-critical scenarios, such as stationary, frontal, and side collisions. The simulator is used to test the responses of AD models to these scenarios, highlighting the limitations of state-of-the-art end-to-end planners in closed-loop settings. While these planners perform well in open-loop scenarios, they fail in safety-critical scenarios, indicating the need for improvements in their safety and real-world usability. The simulator and scenarios are publicly released as an evaluation suite, inviting the research community to test and refine their AD models in realistic environments. The evaluation focuses on collision avoidance rather than displacement metrics, and the results show that two leading end-to-end planners fail severely in safety-critical scenarios despite accurate perception. The simulator uses a closed-loop approach, where the planner predicts a trajectory, which is then executed in a vehicle model, and the results are used to generate realistic sensor data. The evaluation includes quantitative and qualitative results, demonstrating the effectiveness of the simulator in identifying flaws in current AD systems. The study also addresses the real-to-sim gap, showing that the results transfer well to the real world. The simulator is limited in its ability to render certain scenarios and handle deformable objects, but it provides a valuable tool for evaluating the safety of autonomous driving systems. The findings emphasize the importance of improving the safety and robustness of end-to-end planners for real-world deployment.NeuroNCAP is a photorealistic closed-loop simulator for testing autonomous driving (AD) software systems, designed to evaluate safety-critical scenarios. The simulator uses NeRF (Neural Radiance Fields) to generate realistic sensor data from real-world driving logs, enabling the creation of new, unseen scenarios. It allows for reconfiguration and rendering of various safety-critical scenarios, such as stationary, frontal, and side collisions. The simulator is used to test the responses of AD models to these scenarios, highlighting the limitations of state-of-the-art end-to-end planners in closed-loop settings. While these planners perform well in open-loop scenarios, they fail in safety-critical scenarios, indicating the need for improvements in their safety and real-world usability. The simulator and scenarios are publicly released as an evaluation suite, inviting the research community to test and refine their AD models in realistic environments. The evaluation focuses on collision avoidance rather than displacement metrics, and the results show that two leading end-to-end planners fail severely in safety-critical scenarios despite accurate perception. The simulator uses a closed-loop approach, where the planner predicts a trajectory, which is then executed in a vehicle model, and the results are used to generate realistic sensor data. The evaluation includes quantitative and qualitative results, demonstrating the effectiveness of the simulator in identifying flaws in current AD systems. The study also addresses the real-to-sim gap, showing that the results transfer well to the real world. The simulator is limited in its ability to render certain scenarios and handle deformable objects, but it provides a valuable tool for evaluating the safety of autonomous driving systems. The findings emphasize the importance of improving the safety and robustness of end-to-end planners for real-world deployment.