This paper presents a novel approach to optimizing autonomous driving safety by integrating Reinforcement Learning from Human Feedback (RLHF) with Large Language Models (LLMs). The authors, from Rutgers University, propose a multi-agent framework that simulates real-life road environments using pre-trained autonomous car models and human-controlled agents such as cars and pedestrians. The framework leverages both physical and physiological feedback to fine-tune the autonomous car model, with LLMs aiding in interpreting human data and optimizing the RL training loop. The system includes various sensors to collect multimodal data, which is then used to train the autonomous vehicle model. The authors validate their model using data from real-life testbeds in New Jersey and New York City. The initial implementation demonstrates the integration of LLMs with the car simulation system, and future work will focus on evaluating the model's robustness across different types of multimodal models and real-life data. The goal is to develop a safe driving model that can navigate real-life roads and contribute to road safety.This paper presents a novel approach to optimizing autonomous driving safety by integrating Reinforcement Learning from Human Feedback (RLHF) with Large Language Models (LLMs). The authors, from Rutgers University, propose a multi-agent framework that simulates real-life road environments using pre-trained autonomous car models and human-controlled agents such as cars and pedestrians. The framework leverages both physical and physiological feedback to fine-tune the autonomous car model, with LLMs aiding in interpreting human data and optimizing the RL training loop. The system includes various sensors to collect multimodal data, which is then used to train the autonomous vehicle model. The authors validate their model using data from real-life testbeds in New Jersey and New York City. The initial implementation demonstrates the integration of LLMs with the car simulation system, and future work will focus on evaluating the model's robustness across different types of multimodal models and real-life data. The goal is to develop a safe driving model that can navigate real-life roads and contribute to road safety.