This paper introduces a privacy-preserving framework for generating synthetic instructions to align large language models (LLMs) with user intentions, addressing privacy risks associated with collecting and using user instructions. User instructions, which may contain sensitive information, are typically annotated by human workers, posing privacy risks not addressed by standard private optimization. To mitigate this, the authors propose using synthetic instructions generated through differentially private (DP) fine-tuning of language models, which are then resampled to match the distribution of real instructions. A novel filtering algorithm is introduced to align the synthetic instructions' distribution with that of real ones, ensuring high utility in both supervised fine-tuning and reinforcement learning from human feedback. The framework is evaluated on real-world datasets, demonstrating that synthetic instructions generated with DP can achieve comparable performance to real instructions, with models trained on DP synthetic instructions outperforming leading open-source models. The approach ensures strong privacy guarantees by limiting the influence of individual training samples and protecting against empirical attacks. The framework is validated through extensive experiments on publicly available datasets, showing that synthetic instructions can be effectively used for LLM alignment while preserving user privacy. The study highlights the importance of addressing privacy risks in the annotation and training processes of LLMs, particularly in the context of sensitive user data.This paper introduces a privacy-preserving framework for generating synthetic instructions to align large language models (LLMs) with user intentions, addressing privacy risks associated with collecting and using user instructions. User instructions, which may contain sensitive information, are typically annotated by human workers, posing privacy risks not addressed by standard private optimization. To mitigate this, the authors propose using synthetic instructions generated through differentially private (DP) fine-tuning of language models, which are then resampled to match the distribution of real instructions. A novel filtering algorithm is introduced to align the synthetic instructions' distribution with that of real ones, ensuring high utility in both supervised fine-tuning and reinforcement learning from human feedback. The framework is evaluated on real-world datasets, demonstrating that synthetic instructions generated with DP can achieve comparable performance to real instructions, with models trained on DP synthetic instructions outperforming leading open-source models. The approach ensures strong privacy guarantees by limiting the influence of individual training samples and protecting against empirical attacks. The framework is validated through extensive experiments on publicly available datasets, showing that synthetic instructions can be effectively used for LLM alignment while preserving user privacy. The study highlights the importance of addressing privacy risks in the annotation and training processes of LLMs, particularly in the context of sensitive user data.