**PeriodicLoRA (PLoRA): Breaking the Low-Rank Bottleneck in LoRA Optimization** Supervised fine-tuning is a common method to adapt large language models (LLMs) to downstream tasks, but full fine-tuning requires significant computational resources. Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, have gained popularity due to their cost-effectiveness. However, LoRA's performance is limited by the low-rank nature of its weight updates, which can lead to a performance gap compared to full fine-tuning. To address this issue, the authors propose PeriodicLoRA (PLoRA), which accumulates low-rank update matrices multiple times to achieve higher-rank updates. PLoRA involves multiple training stages, where LoRA weights are updated and then unlaunched into the backbone parameters at the end of each stage. This process is repeated, allowing for the accumulation of multiple low-rank updates to form a higher-rank update matrix. Experimental results show that PLoRA outperforms LoRA with approximately 1.8 times the learning ability, without increasing memory usage. Additionally, a momentum-based unloading strategy is introduced to mitigate training instability. The paper includes a detailed analysis of PLoRA's effectiveness, including ablation studies and performance comparisons on various benchmarks. The results demonstrate that PLoRA consistently outperforms LoRA in instruction tuning tasks, showing stronger learning capabilities. The method is also evaluated on challenging benchmarks like GSM8K, MMLU, and ARC, where PLoRA achieves significant improvements over LoRA. **Contributions:** - Introduction of PLoRA for parameter-efficient fine-tuning, breaking the low-rank bottleneck in LoRA. - Validation of PLoRA's effectiveness in instruction tuning, outperforming LoRA. - Analysis of PLoRA's training process and its impact on learning capabilities. - Public sharing of hyperparameter settings to facilitate further research. **Limitations:** - The study primarily evaluates PLoRA in the context of instruction fine-tuning for difficult tasks. - Further research is needed to apply PLoRA to more complex multimodal tasks involving image and text data.**PeriodicLoRA (PLoRA): Breaking the Low-Rank Bottleneck in LoRA Optimization** Supervised fine-tuning is a common method to adapt large language models (LLMs) to downstream tasks, but full fine-tuning requires significant computational resources. Parameter-efficient fine-tuning (PEFT) methods, such as LoRA, have gained popularity due to their cost-effectiveness. However, LoRA's performance is limited by the low-rank nature of its weight updates, which can lead to a performance gap compared to full fine-tuning. To address this issue, the authors propose PeriodicLoRA (PLoRA), which accumulates low-rank update matrices multiple times to achieve higher-rank updates. PLoRA involves multiple training stages, where LoRA weights are updated and then unlaunched into the backbone parameters at the end of each stage. This process is repeated, allowing for the accumulation of multiple low-rank updates to form a higher-rank update matrix. Experimental results show that PLoRA outperforms LoRA with approximately 1.8 times the learning ability, without increasing memory usage. Additionally, a momentum-based unloading strategy is introduced to mitigate training instability. The paper includes a detailed analysis of PLoRA's effectiveness, including ablation studies and performance comparisons on various benchmarks. The results demonstrate that PLoRA consistently outperforms LoRA in instruction tuning tasks, showing stronger learning capabilities. The method is also evaluated on challenging benchmarks like GSM8K, MMLU, and ARC, where PLoRA achieves significant improvements over LoRA. **Contributions:** - Introduction of PLoRA for parameter-efficient fine-tuning, breaking the low-rank bottleneck in LoRA. - Validation of PLoRA's effectiveness in instruction tuning, outperforming LoRA. - Analysis of PLoRA's training process and its impact on learning capabilities. - Public sharing of hyperparameter settings to facilitate further research. **Limitations:** - The study primarily evaluates PLoRA in the context of instruction fine-tuning for difficult tasks. - Further research is needed to apply PLoRA to more complex multimodal tasks involving image and text data.