The paper introduces Federated Matched Averaging (FedMA), a novel algorithm for federated learning of modern neural network architectures such as convolutional neural networks (CNNs) and long short-term memory (LSTMs). FedMA addresses the issue of coordinate-wise averaging in traditional federated learning algorithms, which can lead to poor performance and increased communication overhead. By constructing the shared global model layer-wise, FedMA matches and averages hidden elements with similar feature extraction signatures. The authors demonstrate that FedMA outperforms state-of-the-art federated learning algorithms on deep CNN and LSTM architectures trained on real-world datasets, while also reducing the overall communication burden. The paper includes a detailed discussion of permutation invariance in neural networks and the formulation of practical parameter averaging under this constraint. Experimental results show that FedMA performs well on both homogeneous and heterogeneous data partitions, and it can handle data bias and improve data efficiency. The authors also provide insights into the interpretability of FedMA, showing how it identifies and averages matching groups of convolutional filters.The paper introduces Federated Matched Averaging (FedMA), a novel algorithm for federated learning of modern neural network architectures such as convolutional neural networks (CNNs) and long short-term memory (LSTMs). FedMA addresses the issue of coordinate-wise averaging in traditional federated learning algorithms, which can lead to poor performance and increased communication overhead. By constructing the shared global model layer-wise, FedMA matches and averages hidden elements with similar feature extraction signatures. The authors demonstrate that FedMA outperforms state-of-the-art federated learning algorithms on deep CNN and LSTM architectures trained on real-world datasets, while also reducing the overall communication burden. The paper includes a detailed discussion of permutation invariance in neural networks and the formulation of practical parameter averaging under this constraint. Experimental results show that FedMA performs well on both homogeneous and heterogeneous data partitions, and it can handle data bias and improve data efficiency. The authors also provide insights into the interpretability of FedMA, showing how it identifies and averages matching groups of convolutional filters.