This paper introduces MOCHA, a novel systems-aware optimization method for federated multi-task learning (FML). The authors address the statistical and systems challenges of federated learning by proposing a framework that learns separate models for each node in a distributed network, leveraging multi-task learning (MTL) to capture relationships between tasks. MOCHA is designed to handle high communication costs, stragglers, and fault tolerance in federated settings, and is shown to achieve significant speedups compared to existing methods. The paper first outlines the unique challenges of federated learning, including statistical challenges such as non-IID data and systems challenges such as node heterogeneity and communication bottlenecks. It then presents the contributions of the work, including the natural choice of MTL for handling statistical challenges, the development of MOCHA as a novel method for solving general MTL problems, and the provision of convergence guarantees that consider these systems challenges. The authors propose a federated MTL framework that extends distributed primal-dual optimization methods to handle the unique challenges of federated learning. MOCHA is designed to handle the systems challenges of federated learning by allowing nodes to approximate solutions to their subproblems based on their own capabilities and constraints. This approach enables the method to be robust to stragglers and fault tolerance in federated settings. The paper also presents a convergence analysis of MOCHA, showing that it converges to a stationary solution of the problem under certain assumptions. The authors demonstrate the empirical performance of MOCHA on real-world federated datasets, showing that it significantly outperforms other methods in terms of average prediction error. The method is also shown to be robust to stragglers and fault tolerance in federated settings. The paper concludes with a discussion of the broader implications of the work, noting that while MOCHA is not directly applicable to non-convex deep learning models, there may be natural connections between this approach and "convexified" deep learning models in the context of kernelized federated multi-task learning. The authors also acknowledge the contributions of others in the field and provide references to related work.This paper introduces MOCHA, a novel systems-aware optimization method for federated multi-task learning (FML). The authors address the statistical and systems challenges of federated learning by proposing a framework that learns separate models for each node in a distributed network, leveraging multi-task learning (MTL) to capture relationships between tasks. MOCHA is designed to handle high communication costs, stragglers, and fault tolerance in federated settings, and is shown to achieve significant speedups compared to existing methods. The paper first outlines the unique challenges of federated learning, including statistical challenges such as non-IID data and systems challenges such as node heterogeneity and communication bottlenecks. It then presents the contributions of the work, including the natural choice of MTL for handling statistical challenges, the development of MOCHA as a novel method for solving general MTL problems, and the provision of convergence guarantees that consider these systems challenges. The authors propose a federated MTL framework that extends distributed primal-dual optimization methods to handle the unique challenges of federated learning. MOCHA is designed to handle the systems challenges of federated learning by allowing nodes to approximate solutions to their subproblems based on their own capabilities and constraints. This approach enables the method to be robust to stragglers and fault tolerance in federated settings. The paper also presents a convergence analysis of MOCHA, showing that it converges to a stationary solution of the problem under certain assumptions. The authors demonstrate the empirical performance of MOCHA on real-world federated datasets, showing that it significantly outperforms other methods in terms of average prediction error. The method is also shown to be robust to stragglers and fault tolerance in federated settings. The paper concludes with a discussion of the broader implications of the work, noting that while MOCHA is not directly applicable to non-convex deep learning models, there may be natural connections between this approach and "convexified" deep learning models in the context of kernelized federated multi-task learning. The authors also acknowledge the contributions of others in the field and provide references to related work.