Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text

Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text

2024 | Abhimanyu Hans, Avi Schwarzschild, Valerii Cherapanova, Hamid Kazemi, Aniruddha Saha, Micah Goldblum, Jonas Geiping, Tom Goldstein
The paper introduces *Binoculars*, a novel method for detecting text generated by large language models (LLMs) in the zero-shot setting, where no training data from the LLM source is available. The method leverages the contrast between two closely related language models to achieve high accuracy in distinguishing human-generated and machine-generated text. *Binoculars* calculates the cross-perplexity, a metric that measures how surprising the next token predictions of one model are to another, to identify LLM-generated text. The method is evaluated on various datasets, including News, Creative Writing, and Student Essay, and is shown to achieve over 90% detection accuracy for ChatGPT-generated text with a false positive rate of 0.01%. The paper also discusses the limitations and reliability of *Binoculars* in different scenarios, such as non-native speaker writing, modified prompting strategies, and memorized text. Overall, *Binoculars* demonstrates superior performance compared to existing detectors, making it a valuable tool for combating the misuse of LLMs in various applications.The paper introduces *Binoculars*, a novel method for detecting text generated by large language models (LLMs) in the zero-shot setting, where no training data from the LLM source is available. The method leverages the contrast between two closely related language models to achieve high accuracy in distinguishing human-generated and machine-generated text. *Binoculars* calculates the cross-perplexity, a metric that measures how surprising the next token predictions of one model are to another, to identify LLM-generated text. The method is evaluated on various datasets, including News, Creative Writing, and Student Essay, and is shown to achieve over 90% detection accuracy for ChatGPT-generated text with a false positive rate of 0.01%. The paper also discusses the limitations and reliability of *Binoculars* in different scenarios, such as non-native speaker writing, modified prompting strategies, and memorized text. Overall, *Binoculars* demonstrates superior performance compared to existing detectors, making it a valuable tool for combating the misuse of LLMs in various applications.
Reach us at info@study.space
[slides] Spotting LLMs With Binoculars%3A Zero-Shot Detection of Machine-Generated Text | StudySpace