Fine-tuning Large Language Models for Domain-specific Machine Translation

Fine-tuning Large Language Models for Domain-specific Machine Translation

23 Feb 2024 | Jiawei Zheng, Hanghai Hong, Xiaoli Wang, Jingsong Su, Yonggui Liang, Shikai Wu
This paper proposes a prompt-oriented fine-tuning method, LlamaIT, to enhance the domain-specific machine translation (MT) capabilities of large language models (LLMs). The method addresses challenges such as the need for input translation examples, post-processing, and over-specialization in domain-specific MT. LlamaIT uses a mix-domain dataset constructed with task-specific instructions and incorporates domain-specific bilingual vocabulary to improve translation of rare words. The method is evaluated on both public and self-constructed datasets, showing significant improvements in domain-specific MT performance while preserving zero-shot capabilities. The approach leverages LoRA for efficient fine-tuning, reducing training costs and computational resources. Experiments demonstrate that LlamaIT outperforms existing methods in translation quality and efficiency, particularly in handling rare words and domain-specific terminology. The results indicate that fine-tuning on mix-domain data enhances domain-specific MT capabilities without compromising general-purpose MT performance. The method is effective in both Chinese-to-English and English-to-Chinese translation tasks, and it is efficient in terms of training time and resource usage. The study highlights the importance of prompt-based fine-tuning in improving LLMs for domain-specific MT tasks.This paper proposes a prompt-oriented fine-tuning method, LlamaIT, to enhance the domain-specific machine translation (MT) capabilities of large language models (LLMs). The method addresses challenges such as the need for input translation examples, post-processing, and over-specialization in domain-specific MT. LlamaIT uses a mix-domain dataset constructed with task-specific instructions and incorporates domain-specific bilingual vocabulary to improve translation of rare words. The method is evaluated on both public and self-constructed datasets, showing significant improvements in domain-specific MT performance while preserving zero-shot capabilities. The approach leverages LoRA for efficient fine-tuning, reducing training costs and computational resources. Experiments demonstrate that LlamaIT outperforms existing methods in translation quality and efficiency, particularly in handling rare words and domain-specific terminology. The results indicate that fine-tuning on mix-domain data enhances domain-specific MT capabilities without compromising general-purpose MT performance. The method is effective in both Chinese-to-English and English-to-Chinese translation tasks, and it is efficient in terms of training time and resource usage. The study highlights the importance of prompt-based fine-tuning in improving LLMs for domain-specific MT tasks.
Reach us at info@study.space