Cross-lingual Editing in Multilingual Language Models

Cross-lingual Editing in Multilingual Language Models

3 Feb 2024 | Himanshu Beniwal, Kowsik Nandagopan, Mayank Singh
This paper introduces the cross-lingual model editing (XME) paradigm, where a fact is edited in one language and the update is propagated across other languages. The study evaluates the effectiveness of model editing techniques (METs) in multilingual settings, particularly when languages belong to different script families. Experiments were conducted using BLOOM, mBERT, and XLM-RoBERTa with Latin and Indic scripts. Results show that state-of-the-art METs face performance limitations in cross-lingual settings, especially when languages are from different script families. The research highlights the need for further development of XME techniques to address these challenges. The study also investigates how different architectures store knowledge and how initial fine-tuning language selection affects editing performance. It finds that encoder-only and decoder-only models store knowledge in different layers, and that fine-tuning on certain languages improves editing performance. The study compares hypernetwork-based METs with traditional fine-tuning and finds that hypernetwork-based approaches can achieve higher performance in cross-lingual settings. The research contributes to understanding the effectiveness of METs in multilingual settings and provides insights into the localization of knowledge in language models. The findings suggest that cross-lingual model editing is a promising area for future research.This paper introduces the cross-lingual model editing (XME) paradigm, where a fact is edited in one language and the update is propagated across other languages. The study evaluates the effectiveness of model editing techniques (METs) in multilingual settings, particularly when languages belong to different script families. Experiments were conducted using BLOOM, mBERT, and XLM-RoBERTa with Latin and Indic scripts. Results show that state-of-the-art METs face performance limitations in cross-lingual settings, especially when languages are from different script families. The research highlights the need for further development of XME techniques to address these challenges. The study also investigates how different architectures store knowledge and how initial fine-tuning language selection affects editing performance. It finds that encoder-only and decoder-only models store knowledge in different layers, and that fine-tuning on certain languages improves editing performance. The study compares hypernetwork-based METs with traditional fine-tuning and finds that hypernetwork-based approaches can achieve higher performance in cross-lingual settings. The research contributes to understanding the effectiveness of METs in multilingual settings and provides insights into the localization of knowledge in language models. The findings suggest that cross-lingual model editing is a promising area for future research.
Reach us at info@study.space