12 Apr 2024 | Daocheng Fu1*, Wenjie Lei21*, Licheng Wen1, Pinlong Cai1†, Song Mao1, Min Dou1, Botian Shi1†, Yu Qiao1
The paper introduces LimSim++, an advanced closed-loop simulation platform designed for deploying Multimodal Large Language Models (M)LLMs in autonomous driving. LimSim++ addresses the limitations of existing simulation platforms by providing extended-duration, multi-scenario simulations and supporting continuous learning and improved generalization. The platform enables users to engage in prompt engineering, model evaluation, and framework enhancement, making it a versatile tool for research and practice. The paper also introduces a baseline (M)LLM-driven closed-loop framework validated through quantitative experiments across diverse scenarios. LimSim++ is open-source and available at <https://pjlab-adg.github.io/limsim-plus/>.
- **Introduction**: LimSim++ is the first closed-loop evaluation platform specifically developed for (M)LLM-driven autonomous driving, offering interactive capabilities and advanced features such as collaborative decision-making, interactive replay, and comprehensive multidimensional scenario assessments.
- **System Overview**: LimSim++ consists of a simulation system and an (M)LLM-powered driver agent. It integrates scenario information from SUMO and visual content from CARLA, enabling continuous learning through mechanisms like evaluation, reflection, memory, and tool library.
- **Multimodal Prompt**: LimSim++ supports various types and modalities of prompt inputs, including text and image descriptions, to meet the needs of diverse (M)LLMs.
- **Reasoning & Decision**: The platform supports zero-shot and few-shot driving approaches, handling various control signals and evaluating vehicle behavior decisions based on trajectory analysis.
- **Evaluation**: The evaluation module quantifies and assesses driving performance using metrics such as route completion and driving score, considering factors like ride comfort, driving efficiency, and safety.
- **Reflection & Memory**: LimSim++ introduces a reflection and memory mechanism for continuous learning, enhancing the driver agent's performance by refining decision-making processes and storing refined reasoning outcomes in a vector database.
- **Experiments**: The paper outlines experimental setups and provides baseline results, demonstrating the effectiveness of LimSim++ in various scenarios and comparing the performance of different (M)LLMs. The continuous learning framework is validated through experiments, showing significant improvements in driving proficiency.
The paper proposes LimSim++, an open-source platform for scenario understanding, decision-making, and evaluation in autonomous driving with (M)LLMs. It introduces a baseline (M)LLM-driven closed-loop framework and demonstrates its effectiveness through experiments in various scenarios.The paper introduces LimSim++, an advanced closed-loop simulation platform designed for deploying Multimodal Large Language Models (M)LLMs in autonomous driving. LimSim++ addresses the limitations of existing simulation platforms by providing extended-duration, multi-scenario simulations and supporting continuous learning and improved generalization. The platform enables users to engage in prompt engineering, model evaluation, and framework enhancement, making it a versatile tool for research and practice. The paper also introduces a baseline (M)LLM-driven closed-loop framework validated through quantitative experiments across diverse scenarios. LimSim++ is open-source and available at <https://pjlab-adg.github.io/limsim-plus/>.
- **Introduction**: LimSim++ is the first closed-loop evaluation platform specifically developed for (M)LLM-driven autonomous driving, offering interactive capabilities and advanced features such as collaborative decision-making, interactive replay, and comprehensive multidimensional scenario assessments.
- **System Overview**: LimSim++ consists of a simulation system and an (M)LLM-powered driver agent. It integrates scenario information from SUMO and visual content from CARLA, enabling continuous learning through mechanisms like evaluation, reflection, memory, and tool library.
- **Multimodal Prompt**: LimSim++ supports various types and modalities of prompt inputs, including text and image descriptions, to meet the needs of diverse (M)LLMs.
- **Reasoning & Decision**: The platform supports zero-shot and few-shot driving approaches, handling various control signals and evaluating vehicle behavior decisions based on trajectory analysis.
- **Evaluation**: The evaluation module quantifies and assesses driving performance using metrics such as route completion and driving score, considering factors like ride comfort, driving efficiency, and safety.
- **Reflection & Memory**: LimSim++ introduces a reflection and memory mechanism for continuous learning, enhancing the driver agent's performance by refining decision-making processes and storing refined reasoning outcomes in a vector database.
- **Experiments**: The paper outlines experimental setups and provides baseline results, demonstrating the effectiveness of LimSim++ in various scenarios and comparing the performance of different (M)LLMs. The continuous learning framework is validated through experiments, showing significant improvements in driving proficiency.
The paper proposes LimSim++, an open-source platform for scenario understanding, decision-making, and evaluation in autonomous driving with (M)LLMs. It introduces a baseline (M)LLM-driven closed-loop framework and demonstrates its effectiveness through experiments in various scenarios.