PeriodicLoRA (PLoRA) is a parameter-efficient fine-tuning method that addresses the low-rank bottleneck in LoRA optimization. LoRA, a widely used method for parameter-efficient fine-tuning, limits performance due to its low-rank update matrices. PLoRA improves this by periodically unloading LoRA weights into the backbone parameters and reinitializing them, allowing for higher-rank updates without increasing memory usage. PLoRA consists of multiple training stages where LoRA weights are updated, and at the end of each stage, they are merged into the backbone parameters. This process enables PLoRA to achieve stronger learning ability, approximately 1.8 times that of LoRA, without additional memory overhead. PLoRA also incorporates a momentum-based unloading strategy to enhance training stability. The paper evaluates PLoRA on various tasks, including multi-subject multiple-choice, math reasoning, and language understanding. Results show that PLoRA consistently outperforms LoRA with the same rank, demonstrating its effectiveness in instruction tuning. Ablation studies further confirm that PLoRA's learning ability is significantly improved across different tasks. The method is validated on the LLaMA-7B model, with PLoRA achieving notable improvements on the GSM8K and MMLU benchmarks. PLoRA's approach allows for higher-rank updates, approaching the performance of full fine-tuning while maintaining efficient resource usage. PLoRA's training process involves multiple stages, with each stage contributing to the accumulation of low-rank updates. The method's effectiveness is supported by experimental results showing that PLoRA outperforms LoRA in learning ability and performance on complex tasks. However, PLoRA may overfit in later training stages, a challenge that is addressed through careful parameter tuning and momentum settings. The paper also discusses the impact of learning rate and the number of linear layers applied with LoRA, highlighting the importance of these factors in achieving optimal performance. Overall, PLoRA offers a more efficient and effective approach to parameter-efficient fine-tuning, breaking the low-rank bottleneck and enhancing the learning capabilities of large language models.PeriodicLoRA (PLoRA) is a parameter-efficient fine-tuning method that addresses the low-rank bottleneck in LoRA optimization. LoRA, a widely used method for parameter-efficient fine-tuning, limits performance due to its low-rank update matrices. PLoRA improves this by periodically unloading LoRA weights into the backbone parameters and reinitializing them, allowing for higher-rank updates without increasing memory usage. PLoRA consists of multiple training stages where LoRA weights are updated, and at the end of each stage, they are merged into the backbone parameters. This process enables PLoRA to achieve stronger learning ability, approximately 1.8 times that of LoRA, without additional memory overhead. PLoRA also incorporates a momentum-based unloading strategy to enhance training stability. The paper evaluates PLoRA on various tasks, including multi-subject multiple-choice, math reasoning, and language understanding. Results show that PLoRA consistently outperforms LoRA with the same rank, demonstrating its effectiveness in instruction tuning. Ablation studies further confirm that PLoRA's learning ability is significantly improved across different tasks. The method is validated on the LLaMA-7B model, with PLoRA achieving notable improvements on the GSM8K and MMLU benchmarks. PLoRA's approach allows for higher-rank updates, approaching the performance of full fine-tuning while maintaining efficient resource usage. PLoRA's training process involves multiple stages, with each stage contributing to the accumulation of low-rank updates. The method's effectiveness is supported by experimental results showing that PLoRA outperforms LoRA in learning ability and performance on complex tasks. However, PLoRA may overfit in later training stages, a challenge that is addressed through careful parameter tuning and momentum settings. The paper also discusses the impact of learning rate and the number of linear layers applied with LoRA, highlighting the importance of these factors in achieving optimal performance. Overall, PLoRA offers a more efficient and effective approach to parameter-efficient fine-tuning, breaking the low-rank bottleneck and enhancing the learning capabilities of large language models.