ChatScene is an LLM-based agent that generates safety-critical scenarios for autonomous vehicles (AVs) by first creating textual descriptions of scenarios and then converting them into executable simulations using the Scenic programming language. The agent leverages a comprehensive knowledge retrieval component to translate textual descriptions into domain-specific code snippets, enabling the creation of diverse and complex scenarios within the CARLA simulation environment. The agent first generates text-based scenario descriptions from unstructured language instructions, then breaks these descriptions into sub-descriptions for specific details such as vehicle behaviors and locations. These sub-descriptions are then transformed into domain-specific languages, which generate actual code for prediction and control in simulators. The agent's knowledge retrieval component is trained on a database of scenario description and code pairs, allowing it to efficiently translate specific textual descriptions into corresponding domain-specific code snippets. Extensive experimental results show that ChatScene significantly improves the safety of AVs, with scenarios generated by ChatScene showing a 15% increase in collision rates compared to state-of-the-art baselines when tested against different reinforcement learning-based ego vehicles. Furthermore, using ChatScene-generated scenarios to fine-tune different RL-based autonomous driving models results in a 9% reduction in collision rates, surpassing current SOTA methods. ChatScene effectively bridges the gap between textual descriptions of traffic scenarios and practical CARLA simulations, providing a unified way to conveniently generate safety-critical scenarios for safety testing and improvement for AVs. The code is available at https://github.com/javyduck/ChatScene. The agent's methodology involves a retrieval database of Scenic code snippets, which catalogs diverse adversarial behaviors and traffic configurations, significantly augmenting the variety and critical nature of the driving scenarios generated. In Safebench’s evaluation of eight CARLA Challenge traffic scenarios, our method’s adversarial scenes increased the collision rate by 15% compared to four state-of-the-art (SOTA) baselines, demonstrating the superior safety-critical capabilities of our framework. Subsequent experiments involving the finetuning of the ego vehicle with a subset of our generated adversarial scenarios, followed by comparative evaluations against both the remaining scenarios we created and those from established baselines, demonstrated an additional reduction in average collision rates by at least 9%. Our framework, in conjunction with the retrieval database, not only facilitates direct code generation but also holds potential for future adaptations in multimodal conversions, including text, image, and video, specifically for autonomous driving applications. The experiments show that ChatScene not only elevates collision rates across all base traffic scenarios but also significantly lowers the overall performance scores of ego vehicles. The enhanced scenario diversity also reinforces the effectiveness of our approach. This comprehensive performance underlines the potential of our agent to set new benchmarks in the evaluation and testing of autonomous driving systems.ChatScene is an LLM-based agent that generates safety-critical scenarios for autonomous vehicles (AVs) by first creating textual descriptions of scenarios and then converting them into executable simulations using the Scenic programming language. The agent leverages a comprehensive knowledge retrieval component to translate textual descriptions into domain-specific code snippets, enabling the creation of diverse and complex scenarios within the CARLA simulation environment. The agent first generates text-based scenario descriptions from unstructured language instructions, then breaks these descriptions into sub-descriptions for specific details such as vehicle behaviors and locations. These sub-descriptions are then transformed into domain-specific languages, which generate actual code for prediction and control in simulators. The agent's knowledge retrieval component is trained on a database of scenario description and code pairs, allowing it to efficiently translate specific textual descriptions into corresponding domain-specific code snippets. Extensive experimental results show that ChatScene significantly improves the safety of AVs, with scenarios generated by ChatScene showing a 15% increase in collision rates compared to state-of-the-art baselines when tested against different reinforcement learning-based ego vehicles. Furthermore, using ChatScene-generated scenarios to fine-tune different RL-based autonomous driving models results in a 9% reduction in collision rates, surpassing current SOTA methods. ChatScene effectively bridges the gap between textual descriptions of traffic scenarios and practical CARLA simulations, providing a unified way to conveniently generate safety-critical scenarios for safety testing and improvement for AVs. The code is available at https://github.com/javyduck/ChatScene. The agent's methodology involves a retrieval database of Scenic code snippets, which catalogs diverse adversarial behaviors and traffic configurations, significantly augmenting the variety and critical nature of the driving scenarios generated. In Safebench’s evaluation of eight CARLA Challenge traffic scenarios, our method’s adversarial scenes increased the collision rate by 15% compared to four state-of-the-art (SOTA) baselines, demonstrating the superior safety-critical capabilities of our framework. Subsequent experiments involving the finetuning of the ego vehicle with a subset of our generated adversarial scenarios, followed by comparative evaluations against both the remaining scenarios we created and those from established baselines, demonstrated an additional reduction in average collision rates by at least 9%. Our framework, in conjunction with the retrieval database, not only facilitates direct code generation but also holds potential for future adaptations in multimodal conversions, including text, image, and video, specifically for autonomous driving applications. The experiments show that ChatScene not only elevates collision rates across all base traffic scenarios but also significantly lowers the overall performance scores of ego vehicles. The enhanced scenario diversity also reinforces the effectiveness of our approach. This comprehensive performance underlines the potential of our agent to set new benchmarks in the evaluation and testing of autonomous driving systems.