This paper introduces FlexLoRA, a novel aggregation scheme for federated fine-tuning of Large Language Models (LLMs) that addresses the challenges posed by heterogeneous client resources and data distributions. FlexLoRA dynamically adjusts local LoRA ranks to mitigate the "bucket effect," allowing clients with ample resources to contribute more general knowledge to the global model. By synthesizing full-size LoRA weights from individual client contributions and using Singular Value Decomposition (SVD) for weight redistribution, FlexLoRA fully leverages heterogeneous client resources. Extensive experiments on a large-scale NLP task dataset with thousands of clients validate the efficacy of FlexLoRA, showing consistent improvements over state-of-the-art Federated Learning (FL) methods in downstream NLP tasks. Theoretical analysis and practical insights into the interplay between LoRA ranks, client numbers, and resource distributions further support the effectiveness of FlexLoRA.This paper introduces FlexLoRA, a novel aggregation scheme for federated fine-tuning of Large Language Models (LLMs) that addresses the challenges posed by heterogeneous client resources and data distributions. FlexLoRA dynamically adjusts local LoRA ranks to mitigate the "bucket effect," allowing clients with ample resources to contribute more general knowledge to the global model. By synthesizing full-size LoRA weights from individual client contributions and using Singular Value Decomposition (SVD) for weight redistribution, FlexLoRA fully leverages heterogeneous client resources. Extensive experiments on a large-scale NLP task dataset with thousands of clients validate the efficacy of FlexLoRA, showing consistent improvements over state-of-the-art Federated Learning (FL) methods in downstream NLP tasks. Theoretical analysis and practical insights into the interplay between LoRA ranks, client numbers, and resource distributions further support the effectiveness of FlexLoRA.