AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotators

AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotators

2 Jun 2024 | Jingwei Ni, Minjing Shi, Dominik Stammbach, Mrinmaya Sachan, Elliott Ash, Markus Leippold
The paper "AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotators" addresses the challenges of factual claim detection in automated fact-checking, particularly the inconsistencies in task definitions and the high cost of manual annotation. The authors propose a unified definition of factual claims based on verifiability and introduce AFaCTA, a framework that leverages large language models (LLMs) to assist in the annotation of factual claims. AFaCTA calibrates its annotation confidence through consistency along three predefined reasoning paths, significantly reducing the need for manual annotation. Extensive evaluations on political speech datasets show that AFaCTA can efficiently assist experts in annotating factual claims and training high-quality classifiers, even with or without expert supervision. The paper also introduces PoliClaim, a comprehensive dataset for factual claim detection covering diverse political topics. The contributions include a unified definition of factual claims, the AFaCTA framework, and the PoliClaim dataset. The authors discuss the limitations of their approach, such as the cost of LLM prompts and the need for further research in other domains.The paper "AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotators" addresses the challenges of factual claim detection in automated fact-checking, particularly the inconsistencies in task definitions and the high cost of manual annotation. The authors propose a unified definition of factual claims based on verifiability and introduce AFaCTA, a framework that leverages large language models (LLMs) to assist in the annotation of factual claims. AFaCTA calibrates its annotation confidence through consistency along three predefined reasoning paths, significantly reducing the need for manual annotation. Extensive evaluations on political speech datasets show that AFaCTA can efficiently assist experts in annotating factual claims and training high-quality classifiers, even with or without expert supervision. The paper also introduces PoliClaim, a comprehensive dataset for factual claim detection covering diverse political topics. The contributions include a unified definition of factual claims, the AFaCTA framework, and the PoliClaim dataset. The authors discuss the limitations of their approach, such as the cost of LLM prompts and the need for further research in other domains.
Reach us at info@study.space
Understanding AFaCTA%3A Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotators