LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks This paper proposes LoRA-Flow, a dynamic fusion method for combining existing LoRAs in large language models (LLMs) for generative tasks. Unlike previous methods that use task-level weights for all tokens, LoRA-Flow dynamically adjusts the influence of different LoRAs based on the current context. The fusion weights are determined by a gate module with a small number of parameters, which can be learned with only 200 training examples. Experiments across six generative tasks show that LoRA-Flow consistently outperforms baselines that use task-level fusion weights. The method is effective in tasks with limited annotated data, where different tokens may require different skills. The fusion gate is designed to adapt to the current context, allowing the model to dynamically select the most appropriate LoRAs for each token. The method is compatible with models of various sizes and has shown promising results on larger models. The results indicate that dynamic fusion weights are necessary for effective LoRA combination in generative tasks. The paper also provides an in-depth analysis of the fusion weights across different layers and time steps, demonstrating the effectiveness of the method in handling complex generative tasks. The results show that the fusion weights vary across different model layers and time steps, and that dynamic fusion weights are essential for achieving high performance in generative tasks. The method is also compared with other few-shot fine-tuning baselines and shows superior performance. The paper concludes that LoRA-Flow is a promising approach for dynamic LoRA fusion in generative tasks.LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks This paper proposes LoRA-Flow, a dynamic fusion method for combining existing LoRAs in large language models (LLMs) for generative tasks. Unlike previous methods that use task-level weights for all tokens, LoRA-Flow dynamically adjusts the influence of different LoRAs based on the current context. The fusion weights are determined by a gate module with a small number of parameters, which can be learned with only 200 training examples. Experiments across six generative tasks show that LoRA-Flow consistently outperforms baselines that use task-level fusion weights. The method is effective in tasks with limited annotated data, where different tokens may require different skills. The fusion gate is designed to adapt to the current context, allowing the model to dynamically select the most appropriate LoRAs for each token. The method is compatible with models of various sizes and has shown promising results on larger models. The results indicate that dynamic fusion weights are necessary for effective LoRA combination in generative tasks. The paper also provides an in-depth analysis of the fusion weights across different layers and time steps, demonstrating the effectiveness of the method in handling complex generative tasks. The results show that the fusion weights vary across different model layers and time steps, and that dynamic fusion weights are essential for achieving high performance in generative tasks. The method is also compared with other few-shot fine-tuning baselines and shows superior performance. The paper concludes that LoRA-Flow is a promising approach for dynamic LoRA fusion in generative tasks.