3 Mar 2017 | Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian Goodfellow, Kunal Talwar
This paper introduces a method called Private Aggregation of Teacher Ensembles (PATE) to protect the privacy of sensitive training data in machine learning. PATE combines multiple models trained on disjoint datasets, using them as "teachers" to train a "student" model. The student learns to predict outputs based on noisy votes from the teachers, without directly accessing individual teacher data or parameters. This approach ensures that the student's training does not depend on any single teacher's data, providing strong privacy guarantees through differential privacy.
The PATE method improves upon previous work by applying to any model, including non-convex models like deep neural networks. It achieves state-of-the-art privacy/utility trade-offs on benchmark datasets such as MNIST and SVHN through an improved privacy analysis and semi-supervised learning. The method uses generative adversarial networks (GANs) for semi-supervised learning, which significantly reduces the need for supervision and thus the privacy loss.
The paper evaluates PATE on MNIST and SVHN, achieving a differential privacy bound of (2.04, 10⁻⁵) for MNIST and (8.19, 10⁻⁶) for SVHN, with high accuracy. These results outperform previous differentially private methods. The PATE approach is also shown to be applicable to other model structures and datasets, including medical data.
The paper demonstrates that PATE provides both intuitive and formal privacy guarantees, ensuring that the privacy of training data is protected even if an adversary has access to the student's internal workings. The method is "black-box," meaning it works with any learning algorithm, making it broadly applicable to various machine learning tasks. The results highlight the benefits of combining semi-supervised knowledge transfer with precise, data-dependent privacy analysis, offering strong privacy guarantees while maintaining high utility.This paper introduces a method called Private Aggregation of Teacher Ensembles (PATE) to protect the privacy of sensitive training data in machine learning. PATE combines multiple models trained on disjoint datasets, using them as "teachers" to train a "student" model. The student learns to predict outputs based on noisy votes from the teachers, without directly accessing individual teacher data or parameters. This approach ensures that the student's training does not depend on any single teacher's data, providing strong privacy guarantees through differential privacy.
The PATE method improves upon previous work by applying to any model, including non-convex models like deep neural networks. It achieves state-of-the-art privacy/utility trade-offs on benchmark datasets such as MNIST and SVHN through an improved privacy analysis and semi-supervised learning. The method uses generative adversarial networks (GANs) for semi-supervised learning, which significantly reduces the need for supervision and thus the privacy loss.
The paper evaluates PATE on MNIST and SVHN, achieving a differential privacy bound of (2.04, 10⁻⁵) for MNIST and (8.19, 10⁻⁶) for SVHN, with high accuracy. These results outperform previous differentially private methods. The PATE approach is also shown to be applicable to other model structures and datasets, including medical data.
The paper demonstrates that PATE provides both intuitive and formal privacy guarantees, ensuring that the privacy of training data is protected even if an adversary has access to the student's internal workings. The method is "black-box," meaning it works with any learning algorithm, making it broadly applicable to various machine learning tasks. The results highlight the benefits of combining semi-supervised knowledge transfer with precise, data-dependent privacy analysis, offering strong privacy guarantees while maintaining high utility.