Preference-Conditioned Language-Guided Abstraction (PLGA) is a framework that uses language models (LMs) to infer and incorporate latent human preferences into state abstractions for robot learning. The goal is to construct abstractions that capture how humans perceive and prioritize task-relevant features, enabling more flexible and efficient learning. PLGA leverages LM priors to generate state abstractions that are conditioned on human preferences, which are inferred from behavioral changes in demonstrations. The framework involves two main steps: first, querying the LM to estimate the human preference based on behavioral differences; second, using that preference to construct the state abstraction. The LM can also actively query the human when uncertain about its own estimates. PLGA was tested in simulated experiments, a user study, and on a real Spot robot performing mobile manipulation tasks. The results showed that PLGA outperformed baselines like LGA and GCBC in terms of policy success rates and user interaction experience. In the user study, PLGA was found to be more natural and less effortful for users to specify preferences, leading to better downstream task performance. On the Spot robot, PLGA successfully enabled the robot to complete tasks even when faced with new distractors or unseen linguistic specifications, demonstrating its ability to generalize and adapt to human preferences. The framework addresses the challenge of learning from demonstrations by incorporating human preferences into state abstractions, which are often difficult to specify explicitly. By using LM priors and querying human preferences when necessary, PLGA provides a flexible and effective approach to learning from demonstrations in real-world robotics settings. The results highlight the potential of using language models to generate preference-conditioned state abstractions, which can improve the adaptability and performance of robots in complex tasks.Preference-Conditioned Language-Guided Abstraction (PLGA) is a framework that uses language models (LMs) to infer and incorporate latent human preferences into state abstractions for robot learning. The goal is to construct abstractions that capture how humans perceive and prioritize task-relevant features, enabling more flexible and efficient learning. PLGA leverages LM priors to generate state abstractions that are conditioned on human preferences, which are inferred from behavioral changes in demonstrations. The framework involves two main steps: first, querying the LM to estimate the human preference based on behavioral differences; second, using that preference to construct the state abstraction. The LM can also actively query the human when uncertain about its own estimates. PLGA was tested in simulated experiments, a user study, and on a real Spot robot performing mobile manipulation tasks. The results showed that PLGA outperformed baselines like LGA and GCBC in terms of policy success rates and user interaction experience. In the user study, PLGA was found to be more natural and less effortful for users to specify preferences, leading to better downstream task performance. On the Spot robot, PLGA successfully enabled the robot to complete tasks even when faced with new distractors or unseen linguistic specifications, demonstrating its ability to generalize and adapt to human preferences. The framework addresses the challenge of learning from demonstrations by incorporating human preferences into state abstractions, which are often difficult to specify explicitly. By using LM priors and querying human preferences when necessary, PLGA provides a flexible and effective approach to learning from demonstrations in real-world robotics settings. The results highlight the potential of using language models to generate preference-conditioned state abstractions, which can improve the adaptability and performance of robots in complex tasks.