**Abstract:** This paper introduces RL-VLM-F, a method that automatically generates reward functions for reinforcement learning (RL) agents using only a text description of the task goal and the agent's visual observations. The key innovation is to query vision-language foundation models (VLMs) to provide preferences over pairs of image observations based on the task goal text description, and then learn a reward function from these preference labels. This approach avoids the need for raw reward scores, which can be noisy and inconsistent. The method is evaluated on various tasks, including classic control and manipulation of rigid, articulated, and deformable objects, demonstrating superior performance compared to prior methods that rely on large pre-trained models for reward generation. **Introduction:** Designing effective reward functions is a significant challenge in RL, often requiring extensive human effort and iterative trial-and-error. RL-VLM-F aims to automate this process by leveraging VLMs to generate preferences over image observations, eliminating the need for manual reward function design. Prior work has explored using large language models (LLMs) to write code-based reward functions or extract intrinsic rewards, but these methods often require access to environment code or low-level state information. RL-VLM-F addresses these limitations by using VLMs to compare visual observations, making it applicable to complex tasks involving deformable objects. **Contributions:** - RL-VLM-F automatically generates reward functions for new tasks using only a text description and visual observations. - It successfully learns policies for various manipulation tasks, outperforming prior methods. - Extensive analysis and ablation studies provide insights into the learning procedure and performance gains. **Related Work:** The paper discusses related work in inverse reinforcement learning, learning from human feedback, and using large pre-trained models as reward functions. RL-VLM-F differs from these approaches by leveraging VLMs to generate preferences, reducing the need for human labeling and improving the accuracy of reward signals. **Method:** RL-VLM-F works by initializing a policy and reward function, then iteratively updating the policy using RL and collecting image observations. VLMs are queried to generate preferences over pairs of image observations, which are used to update the reward function. The method is evaluated on tasks ranging from classic control to complex manipulation tasks, showing superior performance compared to baselines. **Experiments:** The paper evaluates RL-VLM-F on seven tasks, including CartPole, rigid and articulated object manipulation, and deformable object manipulation. The results demonstrate that RL-VLM-F outperforms baselines in all tasks, with better alignment with ground-truth task progress and more effective reward functions. **Conclusion:** RL-VLM-F is a novel method that automatically generates reward functions for RL agents using VLMs and preference labels. It shows promise in various manipulation tasks, offering a practical approach to applying RL in real-world settings. Future work could explore active learning and more advanced VLMs to address more complex tasks.**Abstract:** This paper introduces RL-VLM-F, a method that automatically generates reward functions for reinforcement learning (RL) agents using only a text description of the task goal and the agent's visual observations. The key innovation is to query vision-language foundation models (VLMs) to provide preferences over pairs of image observations based on the task goal text description, and then learn a reward function from these preference labels. This approach avoids the need for raw reward scores, which can be noisy and inconsistent. The method is evaluated on various tasks, including classic control and manipulation of rigid, articulated, and deformable objects, demonstrating superior performance compared to prior methods that rely on large pre-trained models for reward generation. **Introduction:** Designing effective reward functions is a significant challenge in RL, often requiring extensive human effort and iterative trial-and-error. RL-VLM-F aims to automate this process by leveraging VLMs to generate preferences over image observations, eliminating the need for manual reward function design. Prior work has explored using large language models (LLMs) to write code-based reward functions or extract intrinsic rewards, but these methods often require access to environment code or low-level state information. RL-VLM-F addresses these limitations by using VLMs to compare visual observations, making it applicable to complex tasks involving deformable objects. **Contributions:** - RL-VLM-F automatically generates reward functions for new tasks using only a text description and visual observations. - It successfully learns policies for various manipulation tasks, outperforming prior methods. - Extensive analysis and ablation studies provide insights into the learning procedure and performance gains. **Related Work:** The paper discusses related work in inverse reinforcement learning, learning from human feedback, and using large pre-trained models as reward functions. RL-VLM-F differs from these approaches by leveraging VLMs to generate preferences, reducing the need for human labeling and improving the accuracy of reward signals. **Method:** RL-VLM-F works by initializing a policy and reward function, then iteratively updating the policy using RL and collecting image observations. VLMs are queried to generate preferences over pairs of image observations, which are used to update the reward function. The method is evaluated on tasks ranging from classic control to complex manipulation tasks, showing superior performance compared to baselines. **Experiments:** The paper evaluates RL-VLM-F on seven tasks, including CartPole, rigid and articulated object manipulation, and deformable object manipulation. The results demonstrate that RL-VLM-F outperforms baselines in all tasks, with better alignment with ground-truth task progress and more effective reward functions. **Conclusion:** RL-VLM-F is a novel method that automatically generates reward functions for RL agents using VLMs and preference labels. It shows promise in various manipulation tasks, offering a practical approach to applying RL in real-world settings. Future work could explore active learning and more advanced VLMs to address more complex tasks.