This paper presents a method called "distillation" to transfer knowledge from a large, complex model to a smaller, more deployable model. The approach involves training a large model (often an ensemble of models) and then using it to train a smaller model by using the large model's outputs as "soft targets." This allows the smaller model to learn the knowledge of the larger model without needing to be as complex. The paper shows that distillation can significantly improve the performance of a smaller model compared to a model trained directly on the same data. For example, on the MNIST dataset, a smaller model trained using distillation achieved better results than a model trained directly on the data. Similarly, in speech recognition, distillation was used to improve the performance of a smaller acoustic model based on a larger model. The distillation process involves training the smaller model to match the output probabilities of the larger model, which are adjusted by increasing the temperature of the softmax function. This makes the probabilities more "soft" and provides more information for the smaller model to learn from. The paper also discusses the use of soft targets to prevent overfitting in specialist models, which are trained on subsets of the data. The paper also compares distillation to other methods, such as mixtures of experts, and shows that distillation can be more efficient and easier to implement. It also discusses the use of distillation in large-scale datasets, such as the JFT dataset, where the method was used to train a large number of specialist models that focus on different subsets of classes. Overall, the paper demonstrates that distillation is a powerful technique for transferring knowledge from a large model to a smaller one, allowing for more efficient and effective deployment of machine learning models.This paper presents a method called "distillation" to transfer knowledge from a large, complex model to a smaller, more deployable model. The approach involves training a large model (often an ensemble of models) and then using it to train a smaller model by using the large model's outputs as "soft targets." This allows the smaller model to learn the knowledge of the larger model without needing to be as complex. The paper shows that distillation can significantly improve the performance of a smaller model compared to a model trained directly on the same data. For example, on the MNIST dataset, a smaller model trained using distillation achieved better results than a model trained directly on the data. Similarly, in speech recognition, distillation was used to improve the performance of a smaller acoustic model based on a larger model. The distillation process involves training the smaller model to match the output probabilities of the larger model, which are adjusted by increasing the temperature of the softmax function. This makes the probabilities more "soft" and provides more information for the smaller model to learn from. The paper also discusses the use of soft targets to prevent overfitting in specialist models, which are trained on subsets of the data. The paper also compares distillation to other methods, such as mixtures of experts, and shows that distillation can be more efficient and easier to implement. It also discusses the use of distillation in large-scale datasets, such as the JFT dataset, where the method was used to train a large number of specialist models that focus on different subsets of classes. Overall, the paper demonstrates that distillation is a powerful technique for transferring knowledge from a large model to a smaller one, allowing for more efficient and effective deployment of machine learning models.