FlexLoRA is a novel aggregation scheme for federated learning (FL) that enhances the fine-tuning of large language models (LLMs) under heterogeneous client resources and tasks. It addresses the "bucket effect" in traditional FL by dynamically adjusting local LoRA ranks, allowing clients with more resources to contribute more global knowledge. FlexLoRA synthesizes a full-size LoRA weight from individual client contributions and uses Singular Value Decomposition (SVD) for weight redistribution, fully leveraging heterogeneous client resources. Experiments involving thousands of clients performing diverse NLP tasks and resource distributions validate FlexLoRA's effectiveness, showing consistent improvements over state-of-the-art FL methods in downstream NLP tasks. Theoretical analysis supports FlexLoRA's practicality and integration with existing LoRA-based FL methods, offering a path toward cross-device, privacy-preserving federated tuning for LLMs. FlexLoRA dynamically adjusts local LoRA ranks based on client resources, enabling diverse LoRA weights across clients. It first forms a low-rank approximation of the LoRA matrix for each client, then computes a weighted average. The resulting global LoRA weight is decomposed using SVD, and the SVD components are redistributed to clients based on their local resources. This process allows clients to receive aggregated knowledge and incorporate it into their local LoRA weights, enhancing the model's generalization ability across diverse data distributions. FlexLoRA maximizes local ranks based on local resources, adhering to the principle of allocating the highest feasible rank given a client's resource budget. Empirical findings show that larger ranks generally yield better generalization. FlexLoRA's lightweight SVD procedure is efficient and scalable, with negligible overhead compared to local LLM training. The method enables heterogeneous ranks without additional hyperparameter tuning, leading to improved convergence rates and overall efficiency. Theoretical analysis shows that FlexLoRA's generalization ability is influenced by the LoRA rank chosen by each client. Increasing the rank improves approximation accuracy, reducing the error bound and the number of samples required for effective generalization. FlexLoRA's effectiveness is further validated through experiments on cross-device FL environments with thousands of clients, demonstrating significant performance improvements in zero-shot generalization and cross-task understanding. Experiments on cross-device FL environments with heterogeneous resource distributions show that FlexLoRA outperforms baselines like FedAvg, FedIT, and SLoRA in generalization and task-specific improvements. FlexLoRA's ability to leverage heterogeneous resource distributions enhances the global model's generalization capability, particularly in heavy-tail-strong resource distributions. The method's scalability is demonstrated through experiments with larger models and diverse tasks, showing improved efficiency and performance in real-world scenarios. FlexLoRA's practicality and effectiveness in cross-device FL settings are supported by both theoretical analysis and empirical results.FlexLoRA is a novel aggregation scheme for federated learning (FL) that enhances the fine-tuning of large language models (LLMs) under heterogeneous client resources and tasks. It addresses the "bucket effect" in traditional FL by dynamically adjusting local LoRA ranks, allowing clients with more resources to contribute more global knowledge. FlexLoRA synthesizes a full-size LoRA weight from individual client contributions and uses Singular Value Decomposition (SVD) for weight redistribution, fully leveraging heterogeneous client resources. Experiments involving thousands of clients performing diverse NLP tasks and resource distributions validate FlexLoRA's effectiveness, showing consistent improvements over state-of-the-art FL methods in downstream NLP tasks. Theoretical analysis supports FlexLoRA's practicality and integration with existing LoRA-based FL methods, offering a path toward cross-device, privacy-preserving federated tuning for LLMs. FlexLoRA dynamically adjusts local LoRA ranks based on client resources, enabling diverse LoRA weights across clients. It first forms a low-rank approximation of the LoRA matrix for each client, then computes a weighted average. The resulting global LoRA weight is decomposed using SVD, and the SVD components are redistributed to clients based on their local resources. This process allows clients to receive aggregated knowledge and incorporate it into their local LoRA weights, enhancing the model's generalization ability across diverse data distributions. FlexLoRA maximizes local ranks based on local resources, adhering to the principle of allocating the highest feasible rank given a client's resource budget. Empirical findings show that larger ranks generally yield better generalization. FlexLoRA's lightweight SVD procedure is efficient and scalable, with negligible overhead compared to local LLM training. The method enables heterogeneous ranks without additional hyperparameter tuning, leading to improved convergence rates and overall efficiency. Theoretical analysis shows that FlexLoRA's generalization ability is influenced by the LoRA rank chosen by each client. Increasing the rank improves approximation accuracy, reducing the error bound and the number of samples required for effective generalization. FlexLoRA's effectiveness is further validated through experiments on cross-device FL environments with thousands of clients, demonstrating significant performance improvements in zero-shot generalization and cross-task understanding. Experiments on cross-device FL environments with heterogeneous resource distributions show that FlexLoRA outperforms baselines like FedAvg, FedIT, and SLoRA in generalization and task-specific improvements. FlexLoRA's ability to leverage heterogeneous resource distributions enhances the global model's generalization capability, particularly in heavy-tail-strong resource distributions. The method's scalability is demonstrated through experiments with larger models and diverse tasks, showing improved efficiency and performance in real-world scenarios. FlexLoRA's practicality and effectiveness in cross-device FL settings are supported by both theoretical analysis and empirical results.