CodecLM is a framework for generating high-quality synthetic data to align large language models (LLMs) with different downstream tasks. It uses LLMs as encoders and decoders to generate instruction metadata, which captures the underlying instruction distribution. This metadata is then used to create tailored instructions, which are further refined using Self-Rubrics and Contrastive Filtering to improve instruction complexity and effectiveness. The framework is evaluated on four open-domain instruction following benchmarks, demonstrating its effectiveness in aligning LLMs with diverse instruction distributions. The key components of CodecLM include encoding seed instructions into metadata, decoding metadata into tailored instructions, and using Self-Rubrics and Contrastive Filtering to refine the instructions. The framework is shown to outperform existing methods in instruction tuning, particularly in generating high-quality instruction-response pairs for different downstream tasks. The results indicate that CodecLM is effective in aligning LLMs with various instruction distributions and can be adapted to different LLMs without human annotation. The framework also demonstrates robustness in handling distribution mismatches and is compatible with various LLMs and data generation techniques. Overall, CodecLM provides a general solution for aligning LLMs with specific instruction distributions and tasks, without the need for human annotation.CodecLM is a framework for generating high-quality synthetic data to align large language models (LLMs) with different downstream tasks. It uses LLMs as encoders and decoders to generate instruction metadata, which captures the underlying instruction distribution. This metadata is then used to create tailored instructions, which are further refined using Self-Rubrics and Contrastive Filtering to improve instruction complexity and effectiveness. The framework is evaluated on four open-domain instruction following benchmarks, demonstrating its effectiveness in aligning LLMs with diverse instruction distributions. The key components of CodecLM include encoding seed instructions into metadata, decoding metadata into tailored instructions, and using Self-Rubrics and Contrastive Filtering to refine the instructions. The framework is shown to outperform existing methods in instruction tuning, particularly in generating high-quality instruction-response pairs for different downstream tasks. The results indicate that CodecLM is effective in aligning LLMs with various instruction distributions and can be adapted to different LLMs without human annotation. The framework also demonstrates robustness in handling distribution mismatches and is compatible with various LLMs and data generation techniques. Overall, CodecLM provides a general solution for aligning LLMs with specific instruction distributions and tasks, without the need for human annotation.