EfficientQAT is a novel quantization-aware training (QAT) technique designed to efficiently compress large language models (LLMs) while maintaining minimal accuracy loss. Traditional QAT methods, such as BitNet b1.58, require extensive training resources and can be impractical for extremely large models. EfficientQAT addresses these challenges by introducing two consecutive phases: Block-wise Training of All Parameters (Block-AP) and End-to-End Training of Quantization Parameters (E2E-QP). 1. **Block-AP**: This phase involves training all parameters in each transformer block using block-wise reconstruction, which reduces memory consumption and avoids training the entire LLM. The process includes quantization and dequantization steps, ensuring precise calibration with full training. 2. **E2E-QP**: This phase fixes the quantized weights and trains only the quantization parameters (step sizes) end-to-end, further enhancing efficiency. This approach significantly reduces the number of trainable parameters, leading to faster convergence and reduced memory usage. Experiments demonstrate that EfficientQAT outperforms existing quantization methods across various models, including base LLMs, instruction-tuned LLMs, and multimodal LLMs, with scales ranging from 7B to 70B parameters. For instance, a 2-bit Llama-2-70B model was trained on a single A100-80GB GPU in 41 hours, achieving less than 3% accuracy degradation compared to full precision. EfficientQAT also shows superior performance in instruction tuning, outperforming existing Q-PEFT methods. The paper includes extensive comparisons with other quantization and fine-tuning techniques, highlighting the effectiveness and efficiency of EfficientQAT. The method is adaptable to different scenarios, including continual pre-training and instruction-tuning, and offers hardware-friendly uniform quantization, making it a promising approach for compressing and accelerating LLMs.EfficientQAT is a novel quantization-aware training (QAT) technique designed to efficiently compress large language models (LLMs) while maintaining minimal accuracy loss. Traditional QAT methods, such as BitNet b1.58, require extensive training resources and can be impractical for extremely large models. EfficientQAT addresses these challenges by introducing two consecutive phases: Block-wise Training of All Parameters (Block-AP) and End-to-End Training of Quantization Parameters (E2E-QP). 1. **Block-AP**: This phase involves training all parameters in each transformer block using block-wise reconstruction, which reduces memory consumption and avoids training the entire LLM. The process includes quantization and dequantization steps, ensuring precise calibration with full training. 2. **E2E-QP**: This phase fixes the quantized weights and trains only the quantization parameters (step sizes) end-to-end, further enhancing efficiency. This approach significantly reduces the number of trainable parameters, leading to faster convergence and reduced memory usage. Experiments demonstrate that EfficientQAT outperforms existing quantization methods across various models, including base LLMs, instruction-tuned LLMs, and multimodal LLMs, with scales ranging from 7B to 70B parameters. For instance, a 2-bit Llama-2-70B model was trained on a single A100-80GB GPU in 41 hours, achieving less than 3% accuracy degradation compared to full precision. EfficientQAT also shows superior performance in instruction tuning, outperforming existing Q-PEFT methods. The paper includes extensive comparisons with other quantization and fine-tuning techniques, highlighting the effectiveness and efficiency of EfficientQAT. The method is adaptable to different scenarios, including continual pre-training and instruction-tuning, and offers hardware-friendly uniform quantization, making it a promising approach for compressing and accelerating LLMs.