Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference

Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference

April 19-23, 2021 | Timo Schick, Hinrich Schütze
This paper introduces Pattern-Exploiting Training (PET), a semi-supervised training method that combines task descriptions with standard supervised learning for few-shot text classification and natural language inference. PET reformulates input examples into cloze-style phrases to help language models understand the task, then uses these phrases to assign soft labels to unlabeled data. Finally, a standard classifier is trained on the resulting soft-labeled dataset. PET outperforms supervised training and strong semi-supervised approaches in low-resource settings. The method is extended to an iterative version, iPET, which trains multiple generations of models on increasingly larger datasets. PET and iPET are evaluated on several tasks in multiple languages, including English and multilingual datasets like x-stance. Results show that PET significantly improves performance, especially when labeled data is limited. PET leverages task-specific human knowledge through pattern-verbalizer pairs, which help the model understand the task better. The method also incorporates auxiliary language modeling to prevent catastrophic forgetting during training. Experiments demonstrate that PET and iPET consistently outperform state-of-the-art semi-supervised methods, highlighting the effectiveness of incorporating human knowledge in the form of PVPs.This paper introduces Pattern-Exploiting Training (PET), a semi-supervised training method that combines task descriptions with standard supervised learning for few-shot text classification and natural language inference. PET reformulates input examples into cloze-style phrases to help language models understand the task, then uses these phrases to assign soft labels to unlabeled data. Finally, a standard classifier is trained on the resulting soft-labeled dataset. PET outperforms supervised training and strong semi-supervised approaches in low-resource settings. The method is extended to an iterative version, iPET, which trains multiple generations of models on increasingly larger datasets. PET and iPET are evaluated on several tasks in multiple languages, including English and multilingual datasets like x-stance. Results show that PET significantly improves performance, especially when labeled data is limited. PET leverages task-specific human knowledge through pattern-verbalizer pairs, which help the model understand the task better. The method also incorporates auxiliary language modeling to prevent catastrophic forgetting during training. Experiments demonstrate that PET and iPET consistently outperform state-of-the-art semi-supervised methods, highlighting the effectiveness of incorporating human knowledge in the form of PVPs.
Reach us at info@study.space