2024 | Herbert Woisetschläger, Alexander Erben, Shiqiang Wang, Ruben Mayer, Hans-Arno Jacobsen
This survey explores efficient federated learning (FL) methods for training foundation models (FMs). FMs are pre-trained on large datasets and can be fine-tuned for specific tasks with smaller data. FL enables privacy-preserving collaborative training across decentralized clients without sharing raw data. However, training large FMs in FL is challenging due to communication and computational costs. This survey introduces a novel taxonomy focusing on computational and communication efficiency for FL applications involving FMs.
The survey identifies three key contributions: (1) a taxonomy of FL methods for FM training, highlighting the gap between computational and communication efficiency techniques; (2) an evaluation of existing FL methods for FM training and communication efficiency; and (3) a discussion of future research directions, emphasizing the need to improve computational and communication efficiency for FL and FMs.
The survey discusses various FL methods for training and fine-tuning FMs, including full model training, parameter-efficient fine-tuning (PEFT), prompt tuning, and instruction tuning. It also explores communication efficiency techniques such as model pruning and full model compression, which reduce the amount of data transmitted between clients and servers.
The survey highlights the importance of efficient communication and training design in FL systems, especially as FMs grow in size. It discusses the challenges of training large FMs in FL, including the need for efficient communication methods and the impact of non-IID data on model performance.
The survey also evaluates the readiness of FL frameworks for FMs, noting that while some frameworks support PEFT techniques like LoRA, many lack efficient communication methods. The survey concludes that further research is needed to improve computational and communication efficiency for FL and FMs, particularly in resource-constrained environments.This survey explores efficient federated learning (FL) methods for training foundation models (FMs). FMs are pre-trained on large datasets and can be fine-tuned for specific tasks with smaller data. FL enables privacy-preserving collaborative training across decentralized clients without sharing raw data. However, training large FMs in FL is challenging due to communication and computational costs. This survey introduces a novel taxonomy focusing on computational and communication efficiency for FL applications involving FMs.
The survey identifies three key contributions: (1) a taxonomy of FL methods for FM training, highlighting the gap between computational and communication efficiency techniques; (2) an evaluation of existing FL methods for FM training and communication efficiency; and (3) a discussion of future research directions, emphasizing the need to improve computational and communication efficiency for FL and FMs.
The survey discusses various FL methods for training and fine-tuning FMs, including full model training, parameter-efficient fine-tuning (PEFT), prompt tuning, and instruction tuning. It also explores communication efficiency techniques such as model pruning and full model compression, which reduce the amount of data transmitted between clients and servers.
The survey highlights the importance of efficient communication and training design in FL systems, especially as FMs grow in size. It discusses the challenges of training large FMs in FL, including the need for efficient communication methods and the impact of non-IID data on model performance.
The survey also evaluates the readiness of FL frameworks for FMs, noting that while some frameworks support PEFT techniques like LoRA, many lack efficient communication methods. The survey concludes that further research is needed to improve computational and communication efficiency for FL and FMs, particularly in resource-constrained environments.