LLM³: Large Language Model-based Task and Motion Planning with Motion Failure Reasoning

LLM³: Large Language Model-based Task and Motion Planning with Motion Failure Reasoning

21 Aug 2024 | Shu Wang, Muzhi Han, Ziyuan Jiao, Zeyu Zhang, Ying Nian Wu, Song-Chun Zhu, Hangxin Liu
LLM³ is a novel Large Language Model (LLM)-based Task and Motion Planning (TAMP) framework that addresses the limitations of traditional TAMP approaches by providing a domain-independent interface between symbolic task planning and continuous motion generation. Traditional TAMP methods rely on manually designed modules to interface between symbolic and continuous domains, which are domain-specific and labor-intensive. LLM³ leverages the powerful reasoning and planning capabilities of pre-trained LLMs to propose symbolic action sequences and select continuous action parameters for motion planning. It incorporates motion planning feedback through prompting, allowing the LLM to iteratively refine its proposals by reasoning about motion failure. This framework interfaces between task planning and motion planning, alleviating the intricate design process of handling domain-specific messages between them. LLM³ is evaluated in a simulated tabletop box-packing task, demonstrating its effectiveness in solving TAMP problems and the efficiency in selecting action parameters. Ablation studies underscore the significant contribution of motion failure reasoning to the success of LLM³. Furthermore, qualitative experiments on a physical manipulator demonstrate the practical applicability of the approach in real-world settings. The framework operates by iteratively generating action sequences with a pre-trained LLM and verifying their feasibility with a motion planner. The LLM uses motion planning feedback to refine its proposals, leading to more efficient and feasible plans. The system diagram shows the interaction between the LLM and the motion planner, with the LLM generating action sequences and the motion planner verifying their feasibility. LLM³ introduces a new TAMP framework that employs a pre-trained LLM as a domain-independent task planner, informed action parameter sampler, and motion failure reasoner. It categorizes and organizes motion planning feedback to efficiently identify and resolve planner-independent motion failures. Comprehensive experiments in both simulation and the real world demonstrate the effectiveness of LLM³ in solving TAMP problems. The framework shows promising potential in addressing previously unspecified tasks and has the potential to improve the efficiency and effectiveness of robotic manipulation tasks, particularly in scenarios where efficient action parameter selection is crucial for improving planning performance.LLM³ is a novel Large Language Model (LLM)-based Task and Motion Planning (TAMP) framework that addresses the limitations of traditional TAMP approaches by providing a domain-independent interface between symbolic task planning and continuous motion generation. Traditional TAMP methods rely on manually designed modules to interface between symbolic and continuous domains, which are domain-specific and labor-intensive. LLM³ leverages the powerful reasoning and planning capabilities of pre-trained LLMs to propose symbolic action sequences and select continuous action parameters for motion planning. It incorporates motion planning feedback through prompting, allowing the LLM to iteratively refine its proposals by reasoning about motion failure. This framework interfaces between task planning and motion planning, alleviating the intricate design process of handling domain-specific messages between them. LLM³ is evaluated in a simulated tabletop box-packing task, demonstrating its effectiveness in solving TAMP problems and the efficiency in selecting action parameters. Ablation studies underscore the significant contribution of motion failure reasoning to the success of LLM³. Furthermore, qualitative experiments on a physical manipulator demonstrate the practical applicability of the approach in real-world settings. The framework operates by iteratively generating action sequences with a pre-trained LLM and verifying their feasibility with a motion planner. The LLM uses motion planning feedback to refine its proposals, leading to more efficient and feasible plans. The system diagram shows the interaction between the LLM and the motion planner, with the LLM generating action sequences and the motion planner verifying their feasibility. LLM³ introduces a new TAMP framework that employs a pre-trained LLM as a domain-independent task planner, informed action parameter sampler, and motion failure reasoner. It categorizes and organizes motion planning feedback to efficiently identify and resolve planner-independent motion failures. Comprehensive experiments in both simulation and the real world demonstrate the effectiveness of LLM³ in solving TAMP problems. The framework shows promising potential in addressing previously unspecified tasks and has the potential to improve the efficiency and effectiveness of robotic manipulation tasks, particularly in scenarios where efficient action parameter selection is crucial for improving planning performance.
Reach us at info@study.space