21 Aug 2024 | Shu Wang1*, Muzhi Han1*, Ziyuan Jiao2*,†, Zeyu Zhang2, Ying Nian Wu1, Song-Chun Zhu2, Hangxin Liu2†
The paper introduces LLM³, a novel Large Language Model (LLM)-based Task and Motion Planning (TAMP) framework that addresses the limitations of traditional TAMP approaches, which rely on manually designed interfaces between symbolic task planning and continuous motion generation. LLM³ leverages the reasoning and planning capabilities of pre-trained LLMs to propose symbolic action sequences and generate continuous action parameters for motion planning. A key feature of LLM³ is its ability to incorporate motion planning feedback through prompting, allowing the LLM to iteratively refine its proposals by reasoning about motion failures. This approach simplifies the domain-specific design process and enhances the efficiency and effectiveness of TAMP solutions.
The paper presents a detailed methodology for LLM³, including the system diagram, reasoning and planning processes, and the synthesis of motion planning feedback. Ablation studies demonstrate the effectiveness of LLM³ in solving TAMP problems, particularly in the context of a box-packing task in simulations. The results show that LLM³ reduces the number of LLM calls and motion planner calls, improving planning success rates. Qualitative experiments on a physical manipulator further validate the practical applicability of LLM³ in real-world settings.
The contributions of LLM³ are threefold: (1) it is the first TAMP framework to use a pre-trained LLM as a domain-independent task planner, informed action parameter sampler, and motion failure reasoner; (2) it categorizes and organizes motion planning feedback to enable efficient identification and resolution of planner-independent motion failures; and (3) it demonstrates comprehensive effectiveness through experiments in both simulation and real-world scenarios.The paper introduces LLM³, a novel Large Language Model (LLM)-based Task and Motion Planning (TAMP) framework that addresses the limitations of traditional TAMP approaches, which rely on manually designed interfaces between symbolic task planning and continuous motion generation. LLM³ leverages the reasoning and planning capabilities of pre-trained LLMs to propose symbolic action sequences and generate continuous action parameters for motion planning. A key feature of LLM³ is its ability to incorporate motion planning feedback through prompting, allowing the LLM to iteratively refine its proposals by reasoning about motion failures. This approach simplifies the domain-specific design process and enhances the efficiency and effectiveness of TAMP solutions.
The paper presents a detailed methodology for LLM³, including the system diagram, reasoning and planning processes, and the synthesis of motion planning feedback. Ablation studies demonstrate the effectiveness of LLM³ in solving TAMP problems, particularly in the context of a box-packing task in simulations. The results show that LLM³ reduces the number of LLM calls and motion planner calls, improving planning success rates. Qualitative experiments on a physical manipulator further validate the practical applicability of LLM³ in real-world settings.
The contributions of LLM³ are threefold: (1) it is the first TAMP framework to use a pre-trained LLM as a domain-independent task planner, informed action parameter sampler, and motion failure reasoner; (2) it categorizes and organizes motion planning feedback to enable efficient identification and resolution of planner-independent motion failures; and (3) it demonstrates comprehensive effectiveness through experiments in both simulation and real-world scenarios.