Plan-Seq-Learn (PSL) is a novel approach that integrates large language models (LLMs) with reinforcement learning (RL) to solve long-horizon robotics tasks. PSL decomposes tasks into a sequence of stages, using LLMs for high-level planning, motion planning for skill sequencing, and RL for learning low-level control strategies. The method leverages the complementary strengths of LLMs and RL: LLMs for abstract planning and internet-scale knowledge, and RL for discovering complex control behaviors. PSL achieves state-of-the-art results on over 25 challenging robotics tasks with up to 10 stages, outperforming existing methods in terms of success rates and sample efficiency. The paper also includes experimental evaluations across four benchmark suites, demonstrating PSL's effectiveness in handling contact-rich interactions, sparse rewards, and long-horizon tasks.Plan-Seq-Learn (PSL) is a novel approach that integrates large language models (LLMs) with reinforcement learning (RL) to solve long-horizon robotics tasks. PSL decomposes tasks into a sequence of stages, using LLMs for high-level planning, motion planning for skill sequencing, and RL for learning low-level control strategies. The method leverages the complementary strengths of LLMs and RL: LLMs for abstract planning and internet-scale knowledge, and RL for discovering complex control behaviors. PSL achieves state-of-the-art results on over 25 challenging robotics tasks with up to 10 stages, outperforming existing methods in terms of success rates and sample efficiency. The paper also includes experimental evaluations across four benchmark suites, demonstrating PSL's effectiveness in handling contact-rich interactions, sparse rewards, and long-horizon tasks.