Long-horizon Locomotion and Manipulation on a Quadrupedal Robot with Large Language Models

Long-horizon Locomotion and Manipulation on a Quadrupedal Robot with Large Language Models

8 Apr 2024 | Yutao Ouyang1*, Jinhan Li2*, Yunfei Li2, Zhongyu Li3, Chao Yu2, Koushil Sreenath3, Yi Wu1,2
The paper presents a large language model (LLM)-based system designed to enhance quadrupedal robots with problem-solving abilities for long-horizon tasks. These tasks require high-level understanding and a broad range of locomotion and manipulation skills. The system integrates multiple LLM agents: a semantic planner, a parameter calculator, and a code generator. The semantic planner generates a high-level plan, the parameter calculator predicts continuous parameters, and the code generator converts the plan into executable robot code. At the low level, reinforcement learning (RL) is used to train motion planning and control policies. The system is tested on challenging tasks such as turning off lights and delivering packages, demonstrating successful multi-step strategies and non-trivial behaviors. The LLM-based reasoning layer successfully figures out strategic solutions, while the low-level policies enable rich environmental interactions. The system achieves over 70% success rate in both simulation and real-world experiments. The paper also includes ablation studies and a detailed analysis of the system's performance and limitations.The paper presents a large language model (LLM)-based system designed to enhance quadrupedal robots with problem-solving abilities for long-horizon tasks. These tasks require high-level understanding and a broad range of locomotion and manipulation skills. The system integrates multiple LLM agents: a semantic planner, a parameter calculator, and a code generator. The semantic planner generates a high-level plan, the parameter calculator predicts continuous parameters, and the code generator converts the plan into executable robot code. At the low level, reinforcement learning (RL) is used to train motion planning and control policies. The system is tested on challenging tasks such as turning off lights and delivering packages, demonstrating successful multi-step strategies and non-trivial behaviors. The LLM-based reasoning layer successfully figures out strategic solutions, while the low-level policies enable rich environmental interactions. The system achieves over 70% success rate in both simulation and real-world experiments. The paper also includes ablation studies and a detailed analysis of the system's performance and limitations.
Reach us at info@study.space