February 26, 2021 | Dong Yin *1, Yudong Chen 13, Kannan Ramchandran ‡1, and Peter Bartlett §1,2
This paper addresses the challenge of robust distributed learning in the presence of Byzantine failures, where some computing units may exhibit abnormal or adversarial behavior. The authors develop two robust distributed gradient descent algorithms: one based on median operations and the other on trimmed mean operations. These algorithms are designed to achieve optimal statistical performance while being robust against Byzantine failures. For strongly convex losses, the trimmed-mean-based algorithm achieves an order-optimal statistical error rate, while the median-based algorithm achieves a similar rate under milder assumptions. Additionally, a one-round algorithm using median operations is proposed, which achieves the same optimal rate for strongly convex quadratic losses. The paper provides rigorous statistical error rate guarantees for these algorithms and compares their performance through experiments, demonstrating their effectiveness in handling Byzantine failures.This paper addresses the challenge of robust distributed learning in the presence of Byzantine failures, where some computing units may exhibit abnormal or adversarial behavior. The authors develop two robust distributed gradient descent algorithms: one based on median operations and the other on trimmed mean operations. These algorithms are designed to achieve optimal statistical performance while being robust against Byzantine failures. For strongly convex losses, the trimmed-mean-based algorithm achieves an order-optimal statistical error rate, while the median-based algorithm achieves a similar rate under milder assumptions. Additionally, a one-round algorithm using median operations is proposed, which achieves the same optimal rate for strongly convex quadratic losses. The paper provides rigorous statistical error rate guarantees for these algorithms and compares their performance through experiments, demonstrating their effectiveness in handling Byzantine failures.