Vision-Language Models (VLMs) provide promptable representations for reinforcement learning (RL). This paper introduces PR2L, a framework that uses VLMs to generate task-relevant semantic features for RL policies. By prompting VLMs with task-specific information, PR2L enables policies to leverage the VLM's internal knowledge and reasoning capabilities to produce grounded, task-specific representations. These representations are then used to train RL policies, which can then perform complex, long-horizon tasks in environments like Minecraft and Habitat. PR2L outperforms policies trained on generic, non-promptable image embeddings and instruction-following methods. It also performs comparably to domain-specific embeddings. The approach uses chain-of-thought prompting to generate representations of common-sense semantic reasoning, improving policy performance in novel scenes by 1.5 times. The framework is evaluated on visually-complex tasks in Minecraft and robot navigation in Habitat, demonstrating the effectiveness of using VLMs for embodied control. The key idea is to use VLMs as promptable representations, which encode semantic features of visual observations based on the VLM's internal knowledge and reasoning. These representations are then used to train RL policies, which can then perform complex tasks. The approach is flexible and can be applied to various domains, including robotics and video games. The paper also discusses the design choices for PR2L, including the use of different VLM layers and the incorporation of auxiliary information in prompts. It highlights the importance of task-specific prompts in eliciting useful representations from VLMs. The results show that PR2L provides a promising way to use VLMs for control tasks, leveraging their ability to reason about image semantics and common-sense knowledge. The paper concludes that PR2L is a valuable method for extracting semantic features from images by prompting VLMs with task context. It opens new directions for using VLMs in control tasks, including the potential for other types of foundation models pre-trained with more sophisticated methods. The framework is shown to be effective in both online and offline RL settings, and its results suggest that it is more sample- and compute-efficient than some existing methods.Vision-Language Models (VLMs) provide promptable representations for reinforcement learning (RL). This paper introduces PR2L, a framework that uses VLMs to generate task-relevant semantic features for RL policies. By prompting VLMs with task-specific information, PR2L enables policies to leverage the VLM's internal knowledge and reasoning capabilities to produce grounded, task-specific representations. These representations are then used to train RL policies, which can then perform complex, long-horizon tasks in environments like Minecraft and Habitat. PR2L outperforms policies trained on generic, non-promptable image embeddings and instruction-following methods. It also performs comparably to domain-specific embeddings. The approach uses chain-of-thought prompting to generate representations of common-sense semantic reasoning, improving policy performance in novel scenes by 1.5 times. The framework is evaluated on visually-complex tasks in Minecraft and robot navigation in Habitat, demonstrating the effectiveness of using VLMs for embodied control. The key idea is to use VLMs as promptable representations, which encode semantic features of visual observations based on the VLM's internal knowledge and reasoning. These representations are then used to train RL policies, which can then perform complex tasks. The approach is flexible and can be applied to various domains, including robotics and video games. The paper also discusses the design choices for PR2L, including the use of different VLM layers and the incorporation of auxiliary information in prompts. It highlights the importance of task-specific prompts in eliciting useful representations from VLMs. The results show that PR2L provides a promising way to use VLMs for control tasks, leveraging their ability to reason about image semantics and common-sense knowledge. The paper concludes that PR2L is a valuable method for extracting semantic features from images by prompting VLMs with task context. It opens new directions for using VLMs in control tasks, including the potential for other types of foundation models pre-trained with more sophisticated methods. The framework is shown to be effective in both online and offline RL settings, and its results suggest that it is more sample- and compute-efficient than some existing methods.