ResLoRA is an improved framework of low-rank adaptation (LoRA) that introduces residual paths during training and eliminates them during inference. This approach enhances training efficiency and performance without adding extra parameters or inference cost. ResLoRA combines residual paths with LoRA, enabling faster convergence and better results on tasks like natural language generation (NLG), natural language understanding (NLU), and text-to-image generation. Experiments show that ResLoRA achieves significant improvements in performance and faster convergence compared to LoRA. The method uses three types of residual structures: input-shortcut, block-shortcut, and middle-shortcut. Merging approaches are designed to convert ResLoRA blocks into LoRA blocks during inference, preserving the original structure and ensuring no additional computational cost. The effectiveness of ResLoRA is validated across various models and tasks, demonstrating its robustness and general applicability. The code for ResLoRA is available at https://github.com/microsoft/LMOps/tree/main/reslora.ResLoRA is an improved framework of low-rank adaptation (LoRA) that introduces residual paths during training and eliminates them during inference. This approach enhances training efficiency and performance without adding extra parameters or inference cost. ResLoRA combines residual paths with LoRA, enabling faster convergence and better results on tasks like natural language generation (NLG), natural language understanding (NLU), and text-to-image generation. Experiments show that ResLoRA achieves significant improvements in performance and faster convergence compared to LoRA. The method uses three types of residual structures: input-shortcut, block-shortcut, and middle-shortcut. Merging approaches are designed to convert ResLoRA blocks into LoRA blocks during inference, preserving the original structure and ensuring no additional computational cost. The effectiveness of ResLoRA is validated across various models and tasks, demonstrating its robustness and general applicability. The code for ResLoRA is available at https://github.com/microsoft/LMOps/tree/main/reslora.