STaR-GATE is a method that teaches language models (LMs) to ask clarifying questions to improve their ability to generate personalized responses. The approach involves an iterative process where a pretrained LM, called the Questioner, interacts with a Roleplayer whose preferences are unknown. The Questioner asks questions to elicit preferences, and the Roleplayer responds. The Questioner is then fine-tuned based on the log probability of generating gold responses, which are generated by an Oracle with access to the Roleplayer's preferences. This process is repeated, allowing the Questioner to improve its questioning strategy over time. The method uses a synthetic dataset of 25,500 unique persona-task prompts to simulate conversations between the Questioner and the Roleplayer. The Questioner is trained to ask questions that increase the probability of generating gold responses. After two iterations of self-improvement, the Questioner asks better questions, allowing it to generate responses that are preferred over responses from the initial model on 72% of tasks. The results indicate that teaching a language model to ask better questions leads to better personalized responses. The STaR-GATE algorithm combines active preference elicitation with a self-improvement loop inspired by STaR. It defines a task setting for improving elicitation, creates a reward function based on the log probability of gold responses, and encourages the LM to use the elicited information while avoiding distribution shift through response regularization. The algorithm is outlined in Algorithm 1, which describes the process of generating conversations, filtering them based on the log probability of gold responses, and fine-tuning the Questioner on the filtered questions and responses. The method is evaluated on a diverse set of everyday tasks, showing that training with STaR-GATE increases both the log-probability of gold responses and win rates compared to the initial model. The results demonstrate that STaR-GATE can significantly enhance a model's ability to engage in effective dialog through targeted questioning. The findings highlight the importance of finetuning on self-generated questions and responses, as opposed to just questions or questions and gold responses. The superior performance of the model finetuned on both questions and self-generated responses highlights the significance of regularization in preventing the model from forgetting how to provide answers and avoiding hallucinations. Overall, the results indicate that teaching a language model to ask better questions can improve its ability to provide personalized responses.STaR-GATE is a method that teaches language models (LMs) to ask clarifying questions to improve their ability to generate personalized responses. The approach involves an iterative process where a pretrained LM, called the Questioner, interacts with a Roleplayer whose preferences are unknown. The Questioner asks questions to elicit preferences, and the Roleplayer responds. The Questioner is then fine-tuned based on the log probability of generating gold responses, which are generated by an Oracle with access to the Roleplayer's preferences. This process is repeated, allowing the Questioner to improve its questioning strategy over time. The method uses a synthetic dataset of 25,500 unique persona-task prompts to simulate conversations between the Questioner and the Roleplayer. The Questioner is trained to ask questions that increase the probability of generating gold responses. After two iterations of self-improvement, the Questioner asks better questions, allowing it to generate responses that are preferred over responses from the initial model on 72% of tasks. The results indicate that teaching a language model to ask better questions leads to better personalized responses. The STaR-GATE algorithm combines active preference elicitation with a self-improvement loop inspired by STaR. It defines a task setting for improving elicitation, creates a reward function based on the log probability of gold responses, and encourages the LM to use the elicited information while avoiding distribution shift through response regularization. The algorithm is outlined in Algorithm 1, which describes the process of generating conversations, filtering them based on the log probability of gold responses, and fine-tuning the Questioner on the filtered questions and responses. The method is evaluated on a diverse set of everyday tasks, showing that training with STaR-GATE increases both the log-probability of gold responses and win rates compared to the initial model. The results demonstrate that STaR-GATE can significantly enhance a model's ability to engage in effective dialog through targeted questioning. The findings highlight the importance of finetuning on self-generated questions and responses, as opposed to just questions or questions and gold responses. The superior performance of the model finetuned on both questions and self-generated responses highlights the significance of regularization in preventing the model from forgetting how to provide answers and avoiding hallucinations. Overall, the results indicate that teaching a language model to ask better questions can improve its ability to provide personalized responses.