LoRA-Pro: Are Low-Rank Adapters Properly Optimized? **Abstract:** Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning method that re-parameterizes the original matrix into the product of two low-rank matrices. Despite its efficiency, LoRA often performs worse than full fine-tuning. This paper introduces LoRA-Pro to bridge this performance gap. We identify that LoRA neglects to approximate the optimization process of full fine-tuning. To address this, we introduce the concept of "equivalent gradient," which characterizes the gradient of the original matrix after low-rank approximation. By minimizing the difference between the equivalent gradient and the gradient from full fine-tuning, we derive optimal closed-form solutions for updating the low-rank matrices. Extensive experiments on natural language processing tasks validate the effectiveness of LoRA-Pro. **Introduction:** LoRA has become a prominent method for parameter-efficient fine-tuning of foundational models. However, it often falls short in performance compared to full fine-tuning. LoRA-Pro aims to bridge this gap by approximating the optimization process of full fine-tuning. The equivalent gradient concept allows us to quantify the discrepancy between LoRA and full fine-tuning, and by minimizing this discrepancy, we derive optimal solutions for the low-rank matrices. **Method:** We revisit LoRA and compare it with full fine-tuning from an optimization perspective. We introduce the equivalent gradient, which is derived from the gradients of matrices \(A\) and \(B\). Our goal is to minimize the difference between the equivalent gradient and the gradient from full fine-tuning. We derive closed-form solutions for the optimal gradients of \(A\) and \(B\), ensuring that the equivalent gradient closely matches the optimization dynamics of full fine-tuning. **Experimental Results:** We evaluate LoRA-Pro on various natural language understanding datasets, including GLUE. The results show that LoRA-Pro achieves higher scores on 3 out of 5 datasets and outperforms standard LoRA with an average margin of 6.72. **Conclusion:** LoRA-Pro effectively bridges the performance gap between LoRA and full fine-tuning by approximating the optimization process. Extensive experiments validate its effectiveness in natural language processing tasks.LoRA-Pro: Are Low-Rank Adapters Properly Optimized? **Abstract:** Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning method that re-parameterizes the original matrix into the product of two low-rank matrices. Despite its efficiency, LoRA often performs worse than full fine-tuning. This paper introduces LoRA-Pro to bridge this performance gap. We identify that LoRA neglects to approximate the optimization process of full fine-tuning. To address this, we introduce the concept of "equivalent gradient," which characterizes the gradient of the original matrix after low-rank approximation. By minimizing the difference between the equivalent gradient and the gradient from full fine-tuning, we derive optimal closed-form solutions for updating the low-rank matrices. Extensive experiments on natural language processing tasks validate the effectiveness of LoRA-Pro. **Introduction:** LoRA has become a prominent method for parameter-efficient fine-tuning of foundational models. However, it often falls short in performance compared to full fine-tuning. LoRA-Pro aims to bridge this gap by approximating the optimization process of full fine-tuning. The equivalent gradient concept allows us to quantify the discrepancy between LoRA and full fine-tuning, and by minimizing this discrepancy, we derive optimal solutions for the low-rank matrices. **Method:** We revisit LoRA and compare it with full fine-tuning from an optimization perspective. We introduce the equivalent gradient, which is derived from the gradients of matrices \(A\) and \(B\). Our goal is to minimize the difference between the equivalent gradient and the gradient from full fine-tuning. We derive closed-form solutions for the optimal gradients of \(A\) and \(B\), ensuring that the equivalent gradient closely matches the optimization dynamics of full fine-tuning. **Experimental Results:** We evaluate LoRA-Pro on various natural language understanding datasets, including GLUE. The results show that LoRA-Pro achieves higher scores on 3 out of 5 datasets and outperforms standard LoRA with an average margin of 6.72. **Conclusion:** LoRA-Pro effectively bridges the performance gap between LoRA and full fine-tuning by approximating the optimization process. Extensive experiments validate its effectiveness in natural language processing tasks.