Agnostic Federated Learning (AFL) is a framework designed to address the limitations of traditional federated learning, where a centralized model is trained on data from multiple clients. In federated learning, the model is often optimized for a uniform distribution over client data, which can lead to biased outcomes. AFL proposes a more robust approach by optimizing the centralized model for any target distribution that is a mixture of the client distributions. This framework naturally incorporates fairness, ensuring that the model performs well across all client distributions. The paper argues that the uniform distribution used in federated learning may not be the optimal target distribution, as it can lead to suboptimal or even detrimental performance in scenarios where the target distribution differs significantly from the uniform distribution. For example, in a scenario where a large population of expensive mobile phones is used by technical users, the model trained on a uniform distribution may not perform well for non-technical users. The AFL framework is shown to provide better performance in terms of fairness and generalization. It is supported by theoretical guarantees, including data-dependent Rademacher complexity bounds, which help define an algorithm for AFL. The paper also presents a fast stochastic optimization algorithm for solving the optimization problem associated with AFL, with convergence guarantees under certain conditions. Beyond federated learning, the AFL framework is applicable to other learning scenarios such as cloud computing, domain adaptation, and drifting, where the training and test distributions do not coincide. The framework allows for the training of models without access to the full training data, making it suitable for scenarios where data privacy is a concern. The paper introduces a learning algorithm for AFL that incorporates regularization terms based on the skewness of the mixture weights. This algorithm is designed to minimize the maximum loss across all possible target distributions, ensuring that the model is robust to distributional shifts. The algorithm is analyzed in terms of its convergence properties, with guarantees for the variance of the stochastic gradients when the loss function is convex. The paper concludes that AFL provides a more robust and fair approach to federated learning, with theoretical guarantees and practical benefits in various learning scenarios.Agnostic Federated Learning (AFL) is a framework designed to address the limitations of traditional federated learning, where a centralized model is trained on data from multiple clients. In federated learning, the model is often optimized for a uniform distribution over client data, which can lead to biased outcomes. AFL proposes a more robust approach by optimizing the centralized model for any target distribution that is a mixture of the client distributions. This framework naturally incorporates fairness, ensuring that the model performs well across all client distributions. The paper argues that the uniform distribution used in federated learning may not be the optimal target distribution, as it can lead to suboptimal or even detrimental performance in scenarios where the target distribution differs significantly from the uniform distribution. For example, in a scenario where a large population of expensive mobile phones is used by technical users, the model trained on a uniform distribution may not perform well for non-technical users. The AFL framework is shown to provide better performance in terms of fairness and generalization. It is supported by theoretical guarantees, including data-dependent Rademacher complexity bounds, which help define an algorithm for AFL. The paper also presents a fast stochastic optimization algorithm for solving the optimization problem associated with AFL, with convergence guarantees under certain conditions. Beyond federated learning, the AFL framework is applicable to other learning scenarios such as cloud computing, domain adaptation, and drifting, where the training and test distributions do not coincide. The framework allows for the training of models without access to the full training data, making it suitable for scenarios where data privacy is a concern. The paper introduces a learning algorithm for AFL that incorporates regularization terms based on the skewness of the mixture weights. This algorithm is designed to minimize the maximum loss across all possible target distributions, ensuring that the model is robust to distributional shifts. The algorithm is analyzed in terms of its convergence properties, with guarantees for the variance of the stochastic gradients when the loss function is convex. The paper concludes that AFL provides a more robust and fair approach to federated learning, with theoretical guarantees and practical benefits in various learning scenarios.