Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Models Zihan Fang, Zheng Lin, Zhe Chen, Xianhao Chen, Yue Gao, Yuguang Fang propose FedPipe, an automated federated pipeline for parameter-efficient fine-tuning of large language models (LLMs). FedPipe addresses the challenges of fine-tuning LLMs in federated learning (FL) settings, where edge servers have heterogeneous computing resources and limited memory. FedPipe identifies important weights for fine-tuning, configures low-rank adapters for each edge server, aggregates local adapters to fine-tune the entire LLM, and quantizes parameters to reduce memory usage. Extensive experiments show that FedPipe improves model training efficiency and achieves higher accuracy than state-of-the-art benchmarks. FedPipe is designed to handle the heterogeneity of edge servers, which often have varying computing and network resources. It uses a mixed-integer linear programming (MILP) optimization problem to model the federated learning process and proposes a two-stage method for identifying important weights and selecting optimal parameters for local training. FedPipe also incorporates a lightweight partial weights aggregation approach to improve communication efficiency. FedPipe's key contributions include being the first automated federated pipeline for fine-tuning LLMs for all downstream tasks, modeling the pipeline as an MILP optimization problem, and developing an effective method to find the best low-rank adaptation structure for each edge server. FedPipe also proposes a fast search algorithm to dynamically identify important parameters for local adapter training and quantizes local models into different quantization bits to increase training efficiency. FedPipe is evaluated on the 20NEWS dataset for text classification and the E2E dataset for natural language generation. It outperforms three classical benchmarks: Vanilla Fine-Tuning (FT), Low-Rank Adaptation (LoRA), and FedAdapter. FedPipe achieves higher accuracy and faster convergence rates, demonstrating its effectiveness in handling heterogeneous computing environments. The framework also reduces the number of trainable parameters, significantly decreasing the computing and communication overheads of fine-tuning LLMs.Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Models Zihan Fang, Zheng Lin, Zhe Chen, Xianhao Chen, Yue Gao, Yuguang Fang propose FedPipe, an automated federated pipeline for parameter-efficient fine-tuning of large language models (LLMs). FedPipe addresses the challenges of fine-tuning LLMs in federated learning (FL) settings, where edge servers have heterogeneous computing resources and limited memory. FedPipe identifies important weights for fine-tuning, configures low-rank adapters for each edge server, aggregates local adapters to fine-tune the entire LLM, and quantizes parameters to reduce memory usage. Extensive experiments show that FedPipe improves model training efficiency and achieves higher accuracy than state-of-the-art benchmarks. FedPipe is designed to handle the heterogeneity of edge servers, which often have varying computing and network resources. It uses a mixed-integer linear programming (MILP) optimization problem to model the federated learning process and proposes a two-stage method for identifying important weights and selecting optimal parameters for local training. FedPipe also incorporates a lightweight partial weights aggregation approach to improve communication efficiency. FedPipe's key contributions include being the first automated federated pipeline for fine-tuning LLMs for all downstream tasks, modeling the pipeline as an MILP optimization problem, and developing an effective method to find the best low-rank adaptation structure for each edge server. FedPipe also proposes a fast search algorithm to dynamically identify important parameters for local adapter training and quantizes local models into different quantization bits to increase training efficiency. FedPipe is evaluated on the 20NEWS dataset for text classification and the E2E dataset for natural language generation. It outperforms three classical benchmarks: Vanilla Fine-Tuning (FT), Low-Rank Adaptation (LoRA), and FedAdapter. FedPipe achieves higher accuracy and faster convergence rates, demonstrating its effectiveness in handling heterogeneous computing environments. The framework also reduces the number of trainable parameters, significantly decreasing the computing and communication overheads of fine-tuning LLMs.