This paper introduces Mixture-of-Subspaces LoRA (MoSLoRA), a parameter-efficient fine-tuning method that enhances the performance of large language models (LLMs) by mixing subspaces in the LoRA framework. LoRA, a widely used method for parameter-efficient fine-tuning, updates the weights of a pre-trained model using low-rank matrices. MoSLoRA extends this by decomposing the LoRA weights into subspaces and mixing them using a learnable mixer, which allows for more flexible and effective adaptation. The method is computationally efficient, easy to implement, and applicable to various models, including large language models, multimodal models, and diffusion models. The key idea of MoSLoRA is to decompose the LoRA weights into subspaces and then mix these subspaces using a learnable mixer. This approach is shown to improve performance on tasks such as commonsense reasoning, visual instruction tuning, and subject-driven text-to-image generation. MoSLoRA outperforms LoRA in these tasks, demonstrating its effectiveness and robustness. The method requires only a small number of additional parameters, making it suitable for low-resource scenarios. The paper also compares MoSLoRA with other parameter-efficient fine-tuning methods, including LoRA, LoKr, LoHa, FLoRA, AdaLoRA, and DoRA. It shows that MoSLoRA consistently outperforms these methods in terms of performance and efficiency. Additionally, MoSLoRA is compatible with quantization methods, making it suitable for deployment in resource-constrained environments. The paper also discusses the relationship between MoSLoRA and Mixture-of-Experts (MoE) methods, highlighting the differences in their approaches to mixing subspaces. MoSLoRA is input-agnostic, while MoE methods are input-specific. MoSLoRA also allows for simultaneous adaptation of all subspaces, whereas MoE methods select top-k experts. The experiments conducted in the paper show that MoSLoRA outperforms LoRA in various tasks, including commonsense reasoning, visual instruction tuning, and subject-driven generation. The method is also shown to be effective in low-resource scenarios, where it requires only a small number of additional parameters and training time. The results indicate that MoSLoRA is a promising method for parameter-efficient fine-tuning of large language models.This paper introduces Mixture-of-Subspaces LoRA (MoSLoRA), a parameter-efficient fine-tuning method that enhances the performance of large language models (LLMs) by mixing subspaces in the LoRA framework. LoRA, a widely used method for parameter-efficient fine-tuning, updates the weights of a pre-trained model using low-rank matrices. MoSLoRA extends this by decomposing the LoRA weights into subspaces and mixing them using a learnable mixer, which allows for more flexible and effective adaptation. The method is computationally efficient, easy to implement, and applicable to various models, including large language models, multimodal models, and diffusion models. The key idea of MoSLoRA is to decompose the LoRA weights into subspaces and then mix these subspaces using a learnable mixer. This approach is shown to improve performance on tasks such as commonsense reasoning, visual instruction tuning, and subject-driven text-to-image generation. MoSLoRA outperforms LoRA in these tasks, demonstrating its effectiveness and robustness. The method requires only a small number of additional parameters, making it suitable for low-resource scenarios. The paper also compares MoSLoRA with other parameter-efficient fine-tuning methods, including LoRA, LoKr, LoHa, FLoRA, AdaLoRA, and DoRA. It shows that MoSLoRA consistently outperforms these methods in terms of performance and efficiency. Additionally, MoSLoRA is compatible with quantization methods, making it suitable for deployment in resource-constrained environments. The paper also discusses the relationship between MoSLoRA and Mixture-of-Experts (MoE) methods, highlighting the differences in their approaches to mixing subspaces. MoSLoRA is input-agnostic, while MoE methods are input-specific. MoSLoRA also allows for simultaneous adaptation of all subspaces, whereas MoE methods select top-k experts. The experiments conducted in the paper show that MoSLoRA outperforms LoRA in various tasks, including commonsense reasoning, visual instruction tuning, and subject-driven generation. The method is also shown to be effective in low-resource scenarios, where it requires only a small number of additional parameters and training time. The results indicate that MoSLoRA is a promising method for parameter-efficient fine-tuning of large language models.