Resilience of Large Language Models for Noisy Instructions

Resilience of Large Language Models for Noisy Instructions

15 Apr 2024 | Bin Wang, Chengwei Wei, Zhengyuan Liu, Geyu Lin, Nancy F. Chen
This paper investigates the resilience of large language models (LLMs) to handle noisy instructions, including automatic speech recognition (ASR) errors, optical character recognition (OCR) errors, grammatical mistakes, typographical errors, and distractive content. The study analyzes how these models perform when exposed to such noise and evaluates the effectiveness of a "re-pass" strategy to correct noisy instructions before processing. The results show that while some LLMs show resistance to certain types of noise, their overall performance significantly suffers. The study also highlights the challenges of correcting noisy instructions, particularly for open-source LLMs. The findings emphasize the need for further research into enhancing model resilience to handle noisy inputs. The study also explores the impact of distractive content on model performance, showing that both cooperative and non-cooperative distractions can lead to performance declines. The "re-pass" strategy is evaluated, and while some models like ChatGPT demonstrate effectiveness in correcting errors, others like Llama-2-7B-Chat show limited success. The study concludes that stronger resilience and correction capabilities are needed for LLMs, especially in handling noise from system integration like ASR and OCR, and in processing user requests under both cooperative and non-cooperative settings. The study also acknowledges limitations, including the difficulty of simulating real-world noise patterns and the focus on English benchmarks without extension to multilingual scenarios. The research underscores the importance of developing models that can effectively handle noisy instructions to improve real-world applications.This paper investigates the resilience of large language models (LLMs) to handle noisy instructions, including automatic speech recognition (ASR) errors, optical character recognition (OCR) errors, grammatical mistakes, typographical errors, and distractive content. The study analyzes how these models perform when exposed to such noise and evaluates the effectiveness of a "re-pass" strategy to correct noisy instructions before processing. The results show that while some LLMs show resistance to certain types of noise, their overall performance significantly suffers. The study also highlights the challenges of correcting noisy instructions, particularly for open-source LLMs. The findings emphasize the need for further research into enhancing model resilience to handle noisy inputs. The study also explores the impact of distractive content on model performance, showing that both cooperative and non-cooperative distractions can lead to performance declines. The "re-pass" strategy is evaluated, and while some models like ChatGPT demonstrate effectiveness in correcting errors, others like Llama-2-7B-Chat show limited success. The study concludes that stronger resilience and correction capabilities are needed for LLMs, especially in handling noise from system integration like ASR and OCR, and in processing user requests under both cooperative and non-cooperative settings. The study also acknowledges limitations, including the difficulty of simulating real-world noise patterns and the focus on English benchmarks without extension to multilingual scenarios. The research underscores the importance of developing models that can effectively handle noisy instructions to improve real-world applications.
Reach us at info@study.space
[slides and audio] Resilience of Large Language Models for Noisy Instructions