LoRA-Pro: Are Low-Rank Adapters Properly Optimized? This paper proposes LoRA-Pro, a novel parameter-efficient fine-tuning method that addresses the performance gap between LoRA and full fine-tuning. LoRA, a popular low-rank adaptation method, re-parameterizes weight changes as the product of two low-rank matrices, reducing the number of trainable parameters. However, LoRA's performance often lags behind full fine-tuning. To bridge this gap, we introduce the concept of "Equivalent Gradient," which quantifies the difference in optimization processes between LoRA and full fine-tuning. By minimizing the discrepancy between the equivalent gradient and the gradient from full fine-tuning, we derive optimal closed-form solutions for updating matrices A and B. These solutions ensure that the equivalent gradient closely matches the optimization dynamics of full fine-tuning, enhancing the effectiveness of LoRA. Extensive experiments on natural language processing tasks validate the effectiveness of LoRA-Pro. The method achieves the highest scores on three out of five datasets and the highest average score across all five datasets. LoRA-Pro outperforms standard LoRA by a margin of 6.72 on average. Theoretical analysis shows that the derived solutions guarantee a decrease in the loss during optimization, ensuring the method's effectiveness. LoRA-Pro not only applies a low-rank approximation to the fine-tuning matrix but also maintains consistency with the optimization of full fine-tuning, enabling more effective fine-tuning. The method is implemented using SGD and AdamW optimizers, with detailed pseudo-codes provided for both. The results demonstrate that LoRA-Pro significantly improves the performance of low-rank adaptation in parameter-efficient fine-tuning.LoRA-Pro: Are Low-Rank Adapters Properly Optimized? This paper proposes LoRA-Pro, a novel parameter-efficient fine-tuning method that addresses the performance gap between LoRA and full fine-tuning. LoRA, a popular low-rank adaptation method, re-parameterizes weight changes as the product of two low-rank matrices, reducing the number of trainable parameters. However, LoRA's performance often lags behind full fine-tuning. To bridge this gap, we introduce the concept of "Equivalent Gradient," which quantifies the difference in optimization processes between LoRA and full fine-tuning. By minimizing the discrepancy between the equivalent gradient and the gradient from full fine-tuning, we derive optimal closed-form solutions for updating matrices A and B. These solutions ensure that the equivalent gradient closely matches the optimization dynamics of full fine-tuning, enhancing the effectiveness of LoRA. Extensive experiments on natural language processing tasks validate the effectiveness of LoRA-Pro. The method achieves the highest scores on three out of five datasets and the highest average score across all five datasets. LoRA-Pro outperforms standard LoRA by a margin of 6.72 on average. Theoretical analysis shows that the derived solutions guarantee a decrease in the loss during optimization, ensuring the method's effectiveness. LoRA-Pro not only applies a low-rank approximation to the fine-tuning matrix but also maintains consistency with the optimization of full fine-tuning, enabling more effective fine-tuning. The method is implemented using SGD and AdamW optimizers, with detailed pseudo-codes provided for both. The results demonstrate that LoRA-Pro significantly improves the performance of low-rank adaptation in parameter-efficient fine-tuning.