1,399 research outputs found
TOBB-ETU at CLEF 2019: Prioritizing claims based on check-worthiness
20th Working Notes of CLEF Conference and Labs of the Evaluation Forum, CLEF ( 2019: Lugano; Switzerland)In recent years, we witnessed an incredible amount of misinformation spread over the Internet. However, it is extremely time consuming to analyze the veracity of every claim made on the Internet. Thus, we urgently need automated systems that can prioritize claims based on their check-worthiness, helping fact-checkers to focus on important claims. In this paper, we present our hybrid approach which combines rule-based and supervised methods for CLEF-2019 Check That! Lab's Check-Worthiness task. Our primary model ranked 9th based on MAP, and 6th based on R-P, P@5, and P@20 metrics in the official evaluation of primary submissions. © 2019 CEUR-WS. All rights reserved
Overview of CheckThat! 2020: Automatic Identification and Verification of Claims in Social Media
We present an overview of the third edition of the CheckThat! Lab at CLEF
2020. The lab featured five tasks in two different languages: English and
Arabic. The first four tasks compose the full pipeline of claim verification in
social media: Task 1 on check-worthiness estimation, Task 2 on retrieving
previously fact-checked claims, Task 3 on evidence retrieval, and Task 4 on
claim verification. The lab is completed with Task 5 on check-worthiness
estimation in political debates and speeches. A total of 67 teams registered to
participate in the lab (up from 47 at CLEF 2019), and 23 of them actually
submitted runs (compared to 14 at CLEF 2019). Most teams used deep neural
networks based on BERT, LSTMs, or CNNs, and achieved sizable improvements over
the baselines on all tasks. Here we describe the tasks setup, the evaluation
results, and a summary of the approaches used by the participants, and we
discuss some lessons learned. Last but not least, we release to the research
community all datasets from the lab as well as the evaluation scripts, which
should enable further research in the important tasks of check-worthiness
estimation and automatic claim verification.Comment: Check-Worthiness Estimation, Fact-Checking, Veracity, Evidence-based
Verification, Detecting Previously Fact-Checked Claims, Social Media
Verification, Computational Journalism, COVID-1
Overview of the CLEF-2018 checkthat! lab on automatic identification and verification of political claims
We present an overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. In its starting year, the lab featured two tasks. Task 1 asked to predict which (potential) claims in a political debate should be prioritized for fact-checking; in particular, given a debate or a political speech, the goal was to produce a ranked list of its sentences based on their worthiness for fact-checking. Task 2 asked to assess whether a given check-worthy claim made by a politician in the context of a debate/speech is factually true, half-true, or false. We offered both tasks in English and in Arabic. In terms of data, for both tasks, we focused on debates from the 2016 US Presidential Campaign, as well as on some speeches during and after the campaign (we also provided translations in Arabic), and we relied on comments and factuality judgments from factcheck.org and snopes.com, which we further refined manually. A total of 30 teams registered to participate in the lab, and 9 of them actually submitted runs. The evaluation results show that the most successful approaches used various neural networks (esp. for Task 1) and evidence retrieval from the Web (esp. for Task 2). We release all datasets, the evaluation scripts, and the submissions by the participants, which should enable further research in both check-worthiness estimation and automatic claim verification
Entity Detection for Check-worthiness Prediction: Glasgow Terrier at CLEF CheckThat! 2019
No abstract available
CheckThat! at CLEF 2020: Enabling the Automatic Identification and Verification of Claims in Social Media
We describe the third edition of the CheckThat! Lab, which is part of the
2020 Cross-Language Evaluation Forum (CLEF). CheckThat! proposes four
complementary tasks and a related task from previous lab editions, offered in
English, Arabic, and Spanish. Task 1 asks to predict which tweets in a Twitter
stream are worth fact-checking. Task 2 asks to determine whether a claim posted
in a tweet can be verified using a set of previously fact-checked claims. Task
3 asks to retrieve text snippets from a given set of Web pages that would be
useful for verifying a target tweet's claim. Task 4 asks to predict the
veracity of a target tweet's claim using a set of Web pages and potentially
useful snippets in them. Finally, the lab offers a fifth task that asks to
predict the check-worthiness of the claims made in English political debates
and speeches. CheckThat! features a full evaluation framework. The evaluation
is carried out using mean average precision or precision at rank k for ranking
tasks, and F1 for classification tasks.Comment: Computational journalism, Check-worthiness, Fact-checking, Veracity,
CLEF-2020 CheckThat! La
Overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims. Task 1: Check-Worthiness
We present an overview of the CLEF-2018 CheckThat! Lab on Automatic Identification and Verification of Political Claims, with focus on Task 1: Check-Worthiness. The task asks to predict which claims in a political debate should be prioritized for fact-checking. In particular, given a debate or a political speech, the goal was to produce a ranked list of its sentences based on their worthiness for fact checking. We offered the task in both English and Arabic, based on debates from the 2016 US Presidential Campaign, as well as on some speeches during and after the campaign. A total of 30 teams registered to participate in the Lab and seven teams actually submitted systems for Task 1. The most successful approaches used by the participants relied on recurrent and multi-layer neural networks, as well as on combinations of distributional representations, on matchings claims' vocabulary against lexicons, and on measures of syntactic dependency. The best systems achieved mean average precision of 0.18 and 0.15 on the English and on the Arabic test datasets, respectively. This leaves large room for further improvement, and thus we release all datasets and the scoring scripts, which should enable further research in check-worthiness estimation
Neural check-worthiness ranking with weak supervision:Finding sentences for fact-checking
Automatic fact-checking systems detect misinformation, such as fake news, by
(i) selecting check-worthy sentences for fact-checking, (ii) gathering related
information to the sentences, and (iii) inferring the factuality of the
sentences. Most prior research on (i) uses hand-crafted features to select
check-worthy sentences, and does not explicitly account for the recent finding
that the top weighted terms in both check-worthy and non-check-worthy sentences
are actually overlapping [15]. Motivated by this, we present a neural
check-worthiness sentence ranking model that represents each word in a sentence
by \textit{both} its embedding (aiming to capture its semantics) and its
syntactic dependencies (aiming to capture its role in modifying the semantics
of other terms in the sentence). Our model is an end-to-end trainable neural
network for check-worthiness ranking, which is trained on large amounts of
unlabelled data through weak supervision. Thorough experimental evaluation
against state of the art baselines, with and without weak supervision, shows
our model to be superior at all times (+13% in MAP and +28% at various
Precision cut-offs from the best baseline with statistical significance).
Empirical analysis of the use of weak supervision, word embedding pretraining
on domain-specific data, and the use of syntactic dependencies of our model
reveals that check-worthy sentences contain notably more identical syntactic
dependencies than non-check-worthy sentences.Comment: 6 page
- …