This paper introduces a framework for aligning large language model (LLM) agents with user preferences through learning from user edits. The proposed method, PRELUDE, infers user preferences based on historical edit data and uses these preferences to generate responses. To implement this, the authors propose CIPHER, an algorithm that leverages the LLM to infer user preferences from edits and aggregates them to generate responses. The method is evaluated on two interactive tasks: summarization and email writing, using a simulated GPT-4 user. CIPHER outperforms several baselines by achieving the lowest edit distance cost while maintaining a low LLM query cost. The results show that the learned preferences are similar to the ground truth latent preferences. The paper also discusses the challenges of learning user preferences, including their complexity, context dependence, and the implicit nature of user edits. The authors conclude that their approach provides a cost-effective and interpretable method for aligning LLM agents with user preferences.This paper introduces a framework for aligning large language model (LLM) agents with user preferences through learning from user edits. The proposed method, PRELUDE, infers user preferences based on historical edit data and uses these preferences to generate responses. To implement this, the authors propose CIPHER, an algorithm that leverages the LLM to infer user preferences from edits and aggregates them to generate responses. The method is evaluated on two interactive tasks: summarization and email writing, using a simulated GPT-4 user. CIPHER outperforms several baselines by achieving the lowest edit distance cost while maintaining a low LLM query cost. The results show that the learned preferences are similar to the ground truth latent preferences. The paper also discusses the challenges of learning user preferences, including their complexity, context dependence, and the implicit nature of user edits. The authors conclude that their approach provides a cost-effective and interpretable method for aligning LLM agents with user preferences.