The paper introduces Mixture of LoRA Experts (MoLE), a novel method for composing multiple trained Low-Rank Adaptation (LoRA) modules to enhance the performance of large pre-trained models across various tasks. LoRA is a popular technique for fine-tuning large models, but the effective composition of multiple LoRAs has been challenging due to issues such as diminished generative capabilities and computational costs. MoLE addresses these challenges by treating each layer of trained LoRAs as distinct experts and implementing hierarchical weight control through learnable gating functions. This approach allows MoLE to dynamically optimize the composition weights based on specific domain objectives, preserving the unique characteristics of individual LoRAs while achieving more effective performance. Extensive experiments in both Natural Language Processing (NLP) and Vision & Language (V&L) domains demonstrate that MoLE outperforms existing LoRA composition methods, providing a more flexible and efficient solution for composing multiple trained LoRAs.The paper introduces Mixture of LoRA Experts (MoLE), a novel method for composing multiple trained Low-Rank Adaptation (LoRA) modules to enhance the performance of large pre-trained models across various tasks. LoRA is a popular technique for fine-tuning large models, but the effective composition of multiple LoRAs has been challenging due to issues such as diminished generative capabilities and computational costs. MoLE addresses these challenges by treating each layer of trained LoRAs as distinct experts and implementing hierarchical weight control through learnable gating functions. This approach allows MoLE to dynamically optimize the composition weights based on specific domain objectives, preserving the unique characteristics of individual LoRAs while achieving more effective performance. Extensive experiments in both Natural Language Processing (NLP) and Vision & Language (V&L) domains demonstrate that MoLE outperforms existing LoRA composition methods, providing a more flexible and efficient solution for composing multiple trained LoRAs.