This paper introduces HETLORA, a novel method for federated fine-tuning of on-device foundation models (ODFMs) that addresses the challenges of data and system heterogeneity. ODFMs are small to medium-sized models that can be deployed on devices but require parameter-efficient methods for fine-tuning. HETLORA uses heterogeneous low-rank approximations (LoRAs) to adapt to varying client capabilities and data complexities. Unlike homogeneous LoRA, which applies the same rank across all clients, HETLORA allows different ranks on different clients, enabling more efficient training and better performance. The method includes rank self-pruning to reduce noise and sparsity-weighted aggregation to improve communication and computation efficiency. Experiments show that HETLORA outperforms homogeneous LoRA and full fine-tuning in terms of training speed, communication efficiency, and final performance. It also demonstrates that smaller ranks can prevent overfitting while maintaining good performance. The results indicate that HETLORA is a practical and effective solution for federated fine-tuning of ODFMs in heterogeneous environments.This paper introduces HETLORA, a novel method for federated fine-tuning of on-device foundation models (ODFMs) that addresses the challenges of data and system heterogeneity. ODFMs are small to medium-sized models that can be deployed on devices but require parameter-efficient methods for fine-tuning. HETLORA uses heterogeneous low-rank approximations (LoRAs) to adapt to varying client capabilities and data complexities. Unlike homogeneous LoRA, which applies the same rank across all clients, HETLORA allows different ranks on different clients, enabling more efficient training and better performance. The method includes rank self-pruning to reduce noise and sparsity-weighted aggregation to improve communication and computation efficiency. Experiments show that HETLORA outperforms homogeneous LoRA and full fine-tuning in terms of training speed, communication efficiency, and final performance. It also demonstrates that smaller ranks can prevent overfitting while maintaining good performance. The results indicate that HETLORA is a practical and effective solution for federated fine-tuning of ODFMs in heterogeneous environments.