The paper introduces a novel quantization framework called *ApiQ* designed to improve the performance of memory-efficient fine-tuning of large language models (LLMs). ApiQ addresses the issue of inconsistent performance across different bit-width quantizations and multifaceted tasks, which is a common problem with current strategies like QLoRA. The key innovation of ApiQ is its approach to restoring lost information during quantization by concurrently initializing the LoRA components and quantizing the LLM's weights. This ensures that the original activation precision of the LLM is maintained while mitigating error propagation from shallower to deeper layers. Through comprehensive evaluations on various language tasks and LLMs, ApiQ demonstrates superior performance in terms of activation error minimization and consistent finetuning results across different bit-widths. The paper also discusses the challenges associated with quantization and the effectiveness of ApiQ in reducing quantization errors, making it a valuable contribution to the field of memory-efficient LLM fine-tuning.The paper introduces a novel quantization framework called *ApiQ* designed to improve the performance of memory-efficient fine-tuning of large language models (LLMs). ApiQ addresses the issue of inconsistent performance across different bit-width quantizations and multifaceted tasks, which is a common problem with current strategies like QLoRA. The key innovation of ApiQ is its approach to restoring lost information during quantization by concurrently initializing the LoRA components and quantizing the LLM's weights. This ensures that the original activation precision of the LLM is maintained while mitigating error propagation from shallower to deeper layers. Through comprehensive evaluations on various language tasks and LLMs, ApiQ demonstrates superior performance in terms of activation error minimization and consistent finetuning results across different bit-widths. The paper also discusses the challenges associated with quantization and the effectiveness of ApiQ in reducing quantization errors, making it a valuable contribution to the field of memory-efficient LLM fine-tuning.