AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotators With the rise of generative AI, automated fact-checking methods to combat misinformation are becoming increasingly important. However, factual claim detection, the first step in a fact-checking pipeline, faces two key challenges: (1) inconsistent definitions of the task and what a claim is, and (2) the high cost of manual annotation. To address (1), the authors propose a unifying definition of factual claims based on verifiability. To address (2), they introduce AFaCTA, a novel framework that assists in the annotation of factual claims using large language models (LLMs). AFaCTA calibrates its annotation confidence with consistency along three predefined reasoning paths. Extensive evaluation in the domain of political speech shows that AFaCTA can efficiently assist experts in annotating factual claims and training high-quality classifiers, and can work with or without expert supervision. The analyses also result in PoliClaim, a comprehensive claim detection dataset spanning diverse political topics. The paper discusses the challenges in defining factual claims, including discrepancies in task and claim definitions, and the high cost of manual annotations. The authors propose a definition of factual claims based on verifiability, focusing on statements that can be objectively verified. They also introduce AFaCTA, an LLM-based framework that assists in factual claim annotation by leveraging LLMs to improve annotation quality through consistency along different reasoning paths. The framework includes three prompting steps and an aggregation step, inspired by Kahneman's fast and slow thinking patterns. The authors evaluate AFaCTA's performance on the PoliClaim dataset, which consists of political speeches from 1998 to 2021. They find that AFaCTA outperforms human experts on perfectly consistent samples but performs worse on inconsistent samples. They also test AFaCTA's performance on different domains, including social media, and find that it can be used to annotate factual claims in various domains. The results show that AFaCTA's annotated data with perfect consistency can be a strong substitute for data annotated by human experts. The paper concludes that AFaCTA is a reliable tool for assisting factual claim annotation, with the potential to reduce manual effort and improve the scalability and generalizability of fact-checking systems.AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotators With the rise of generative AI, automated fact-checking methods to combat misinformation are becoming increasingly important. However, factual claim detection, the first step in a fact-checking pipeline, faces two key challenges: (1) inconsistent definitions of the task and what a claim is, and (2) the high cost of manual annotation. To address (1), the authors propose a unifying definition of factual claims based on verifiability. To address (2), they introduce AFaCTA, a novel framework that assists in the annotation of factual claims using large language models (LLMs). AFaCTA calibrates its annotation confidence with consistency along three predefined reasoning paths. Extensive evaluation in the domain of political speech shows that AFaCTA can efficiently assist experts in annotating factual claims and training high-quality classifiers, and can work with or without expert supervision. The analyses also result in PoliClaim, a comprehensive claim detection dataset spanning diverse political topics. The paper discusses the challenges in defining factual claims, including discrepancies in task and claim definitions, and the high cost of manual annotations. The authors propose a definition of factual claims based on verifiability, focusing on statements that can be objectively verified. They also introduce AFaCTA, an LLM-based framework that assists in factual claim annotation by leveraging LLMs to improve annotation quality through consistency along different reasoning paths. The framework includes three prompting steps and an aggregation step, inspired by Kahneman's fast and slow thinking patterns. The authors evaluate AFaCTA's performance on the PoliClaim dataset, which consists of political speeches from 1998 to 2021. They find that AFaCTA outperforms human experts on perfectly consistent samples but performs worse on inconsistent samples. They also test AFaCTA's performance on different domains, including social media, and find that it can be used to annotate factual claims in various domains. The results show that AFaCTA's annotated data with perfect consistency can be a strong substitute for data annotated by human experts. The paper concludes that AFaCTA is a reliable tool for assisting factual claim annotation, with the potential to reduce manual effort and improve the scalability and generalizability of fact-checking systems.