The paper introduces NeuroNCAP, a versatile NeRF-based simulator designed for testing autonomous driving (AD) software systems, focusing on sensor-realistic closed-loop evaluation and the creation of safety-critical scenarios inspired by the European New Car Assessment Programme (Euro NCAP). The simulator learns from real-world driving sensor data and enables the generation of new, unseen scenarios. The evaluation reveals that while state-of-the-art end-to-end planners excel in nominal driving scenarios in an open-loop setting, they exhibit critical flaws when navigating safety-critical scenarios in a closed-loop setting. The authors release their simulator and scenarios as an open-source evaluation suite to encourage research in controlled, yet challenging, sensor-realistic environments. The contributions include an open-source framework for photorealistic closed-loop simulation, the construction of safety-critical scenarios, a novel evaluation protocol focusing on collisions, and the demonstration that two state-of-the-art end-to-end planners fail severely in these scenarios. The paper also discusses related work, methodological details, experimental setup, and limitations, highlighting the need for advancements in the safety and real-world usability of end-to-end planners.The paper introduces NeuroNCAP, a versatile NeRF-based simulator designed for testing autonomous driving (AD) software systems, focusing on sensor-realistic closed-loop evaluation and the creation of safety-critical scenarios inspired by the European New Car Assessment Programme (Euro NCAP). The simulator learns from real-world driving sensor data and enables the generation of new, unseen scenarios. The evaluation reveals that while state-of-the-art end-to-end planners excel in nominal driving scenarios in an open-loop setting, they exhibit critical flaws when navigating safety-critical scenarios in a closed-loop setting. The authors release their simulator and scenarios as an open-source evaluation suite to encourage research in controlled, yet challenging, sensor-realistic environments. The contributions include an open-source framework for photorealistic closed-loop simulation, the construction of safety-critical scenarios, a novel evaluation protocol focusing on collisions, and the demonstration that two state-of-the-art end-to-end planners fail severely in these scenarios. The paper also discusses related work, methodological details, experimental setup, and limitations, highlighting the need for advancements in the safety and real-world usability of end-to-end planners.