SEMI-SUPERVISED KNOWLEDGE TRANSFER FOR DEEP LEARNING FROM PRIVATE TRAINING DATA

SEMI-SUPERVISED KNOWLEDGE TRANSFER FOR DEEP LEARNING FROM PRIVATE TRAINING DATA

3 Mar 2017 | Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian Goodfellow, Kunal Talwar
The paper introduces a method called *Private Aggregation of Teacher Ensembles* (PATE) to protect the privacy of sensitive training data in machine learning applications. PATE combines multiple models trained on disjoint datasets, known as "teacher" models, to create a "student" model. The teacher models are used to predict outputs through noisy voting, ensuring that the student model does not directly access individual teacher models or their underlying data. This approach provides strong privacy guarantees, both intuitively and formally through differential privacy, even if an adversary can query the student model and inspect its internal workings. The paper demonstrates that PATE imposes weak assumptions on the training of teachers, making it applicable to any model, including non-convex models like deep neural networks (DNNs). It achieves state-of-the-art privacy/utility trade-offs on datasets such as MNIST and SVHN through an improved privacy analysis and semi-supervised learning techniques. The use of generative adversarial networks (GANs) further enhances the semi-supervised learning process, improving the accuracy of the student model while maintaining strong privacy guarantees. The privacy analysis of PATE is based on the moments accountant technique, which allows for a precise and data-dependent bound on the privacy cost. The paper also discusses the trade-offs between the number of teachers and the privacy cost, showing that a larger number of teachers can reduce the privacy cost but may decrease the accuracy of individual teacher models. Overall, PATE offers a general and effective approach to protecting the privacy of sensitive training data in machine learning, while maintaining high utility in model performance.The paper introduces a method called *Private Aggregation of Teacher Ensembles* (PATE) to protect the privacy of sensitive training data in machine learning applications. PATE combines multiple models trained on disjoint datasets, known as "teacher" models, to create a "student" model. The teacher models are used to predict outputs through noisy voting, ensuring that the student model does not directly access individual teacher models or their underlying data. This approach provides strong privacy guarantees, both intuitively and formally through differential privacy, even if an adversary can query the student model and inspect its internal workings. The paper demonstrates that PATE imposes weak assumptions on the training of teachers, making it applicable to any model, including non-convex models like deep neural networks (DNNs). It achieves state-of-the-art privacy/utility trade-offs on datasets such as MNIST and SVHN through an improved privacy analysis and semi-supervised learning techniques. The use of generative adversarial networks (GANs) further enhances the semi-supervised learning process, improving the accuracy of the student model while maintaining strong privacy guarantees. The privacy analysis of PATE is based on the moments accountant technique, which allows for a precise and data-dependent bound on the privacy cost. The paper also discusses the trade-offs between the number of teachers and the privacy cost, showing that a larger number of teachers can reduce the privacy cost but may decrease the accuracy of individual teacher models. Overall, PATE offers a general and effective approach to protecting the privacy of sensitive training data in machine learning, while maintaining high utility in model performance.
Reach us at info@study.space