Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation

Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation

November 16–20, 2020 | Nils Reimers and Iryna Gurevych
The paper presents a method to extend monolingual sentence embedding models to new languages using multilingual knowledge distillation. The approach involves training a student model on translated sentences to mimic the behavior of a teacher model in the source language. This method is advantageous because it requires fewer samples to extend models to new languages, ensures desired properties in the vector space, and has lower hardware requirements compared to other methods. The authors demonstrate the effectiveness of their approach for over 50 languages from various families, showing significant improvements in accuracy, especially for low-resource languages. The code for extending sentence embedding models to more than 400 languages is publicly available. The paper also evaluates the impact of different training datasets and compares the approach to existing methods, highlighting its ability to create well-aligned vector spaces across languages and reduce language bias.The paper presents a method to extend monolingual sentence embedding models to new languages using multilingual knowledge distillation. The approach involves training a student model on translated sentences to mimic the behavior of a teacher model in the source language. This method is advantageous because it requires fewer samples to extend models to new languages, ensures desired properties in the vector space, and has lower hardware requirements compared to other methods. The authors demonstrate the effectiveness of their approach for over 50 languages from various families, showing significant improvements in accuracy, especially for low-resource languages. The code for extending sentence embedding models to more than 400 languages is publicly available. The paper also evaluates the impact of different training datasets and compares the approach to existing methods, highlighting its ability to create well-aligned vector spaces across languages and reduce language bias.
Reach us at info@study.space