QuIP# is a post-training quantization (PTQ) method that achieves state-of-the-art results in extreme compression regimes, particularly for large language models (LLMs). It introduces three key techniques: incoherence processing with the randomized Hadamard transform (RHT), lattice-based codebooks derived from the E₈ lattice, and inter-layer fine-tuning. Incoherence processing improves the distribution of weight matrices, enabling more efficient quantization. The E₈ lattice-based codebook achieves high packing density and allows for fast inference. Inter-layer fine-tuning further enhances quantization fidelity. QuIP# outperforms existing PTQ methods, including OmniQuant and AQLM, and enables faster inference. It supports 3-bit models that scale better than theoretically lossless 4-bit models, challenging previous claims about 4-bit optimality. QuIP# is designed for fast inference, achieving over 50% of peak memory bandwidth on a NVIDIA RTX 4090. The method is implemented in Python and available at https://github.com/Cornell-RelaxML/quip-sharp. The paper also discusses the impact of different quantization techniques on model performance and inference speed, highlighting the benefits of using structured codebooks and efficient inference strategies.QuIP# is a post-training quantization (PTQ) method that achieves state-of-the-art results in extreme compression regimes, particularly for large language models (LLMs). It introduces three key techniques: incoherence processing with the randomized Hadamard transform (RHT), lattice-based codebooks derived from the E₈ lattice, and inter-layer fine-tuning. Incoherence processing improves the distribution of weight matrices, enabling more efficient quantization. The E₈ lattice-based codebook achieves high packing density and allows for fast inference. Inter-layer fine-tuning further enhances quantization fidelity. QuIP# outperforms existing PTQ methods, including OmniQuant and AQLM, and enables faster inference. It supports 3-bit models that scale better than theoretically lossless 4-bit models, challenging previous claims about 4-bit optimality. QuIP# is designed for fast inference, achieving over 50% of peak memory bandwidth on a NVIDIA RTX 4090. The method is implemented in Python and available at https://github.com/Cornell-RelaxML/quip-sharp. The paper also discusses the impact of different quantization techniques on model performance and inference speed, highlighting the benefits of using structured codebooks and efficient inference strategies.