Driving Everywhere with Large Language Model Policy Adaptation

Driving Everywhere with Large Language Model Policy Adaptation

10 Apr 2024 | Boyi Li, Yue Wang, Jiageng Mao, Boris Ivanovic, Sushant Veer, Karen Leung, Marco Pavone
The paper introduces LLaDA, a novel framework that enables human drivers and autonomous vehicles (AVs) to adapt their driving behavior to new environments by leveraging the zero-shot generalizability of large language models (LLMs). LLaDA addresses the challenge of adapting to varying traffic rules and customs across different geographical regions, which has been a significant barrier to the widespread deployment of AVs. The system consists of three main components: generating an initial executable policy, using a Traffic Rule Extractor (TRE) to extract relevant traffic rules from local traffic codes, and adapting the plan using a pre-trained LLM (GPT-4V). Extensive user studies and experiments on the nuScenes dataset demonstrate that LLaDA effectively disambiguates unexpected situations and improves AV motion planning policies. The paper also discusses the limitations of LLaDA, such as its sensitivity to the quality of scene descriptions, and outlines future work directions, including improving scene descriptions and developing an unexpected scenario detector. Overall, LLaDA shows promise in enabling AVs to operate more safely and effectively in diverse environments.The paper introduces LLaDA, a novel framework that enables human drivers and autonomous vehicles (AVs) to adapt their driving behavior to new environments by leveraging the zero-shot generalizability of large language models (LLMs). LLaDA addresses the challenge of adapting to varying traffic rules and customs across different geographical regions, which has been a significant barrier to the widespread deployment of AVs. The system consists of three main components: generating an initial executable policy, using a Traffic Rule Extractor (TRE) to extract relevant traffic rules from local traffic codes, and adapting the plan using a pre-trained LLM (GPT-4V). Extensive user studies and experiments on the nuScenes dataset demonstrate that LLaDA effectively disambiguates unexpected situations and improves AV motion planning policies. The paper also discusses the limitations of LLaDA, such as its sensitivity to the quality of scene descriptions, and outlines future work directions, including improving scene descriptions and developing an unexpected scenario detector. Overall, LLaDA shows promise in enabling AVs to operate more safely and effectively in diverse environments.
Reach us at info@study.space
[slides and audio] Driving Everywhere with Large Language Model Policy Adaptation