10 research outputs found
Unsupervised Semantic Hashing with Pairwise Reconstruction
Semantic Hashing is a popular family of methods for efficient similarity
search in large-scale datasets. In Semantic Hashing, documents are encoded as
short binary vectors (i.e., hash codes), such that semantic similarity can be
efficiently computed using the Hamming distance. Recent state-of-the-art
approaches have utilized weak supervision to train better performing hashing
models. Inspired by this, we present Semantic Hashing with Pairwise
Reconstruction (PairRec), which is a discrete variational autoencoder based
hashing model. PairRec first encodes weakly supervised training pairs (a query
document and a semantically similar document) into two hash codes, and then
learns to reconstruct the same query document from both of these hash codes
(i.e., pairwise reconstruction). This pairwise reconstruction enables our model
to encode local neighbourhood structures within the hash code directly through
the decoder. We experimentally compare PairRec to traditional and
state-of-the-art approaches, and obtain significant performance improvements in
the task of document similarity search.Comment: Accepted at SIGIR'2
CheckThat! at CLEF 2020: Enabling the Automatic Identification and Verification of Claims in Social Media
We describe the third edition of the CheckThat! Lab, which is part of the
2020 Cross-Language Evaluation Forum (CLEF). CheckThat! proposes four
complementary tasks and a related task from previous lab editions, offered in
English, Arabic, and Spanish. Task 1 asks to predict which tweets in a Twitter
stream are worth fact-checking. Task 2 asks to determine whether a claim posted
in a tweet can be verified using a set of previously fact-checked claims. Task
3 asks to retrieve text snippets from a given set of Web pages that would be
useful for verifying a target tweet's claim. Task 4 asks to predict the
veracity of a target tweet's claim using a set of Web pages and potentially
useful snippets in them. Finally, the lab offers a fifth task that asks to
predict the check-worthiness of the claims made in English political debates
and speeches. CheckThat! features a full evaluation framework. The evaluation
is carried out using mean average precision or precision at rank k for ranking
tasks, and F1 for classification tasks.Comment: Computational journalism, Check-worthiness, Fact-checking, Veracity,
CLEF-2020 CheckThat! La
Denmark's Participation in the Search Engine TREC COVID-19 Challenge: Lessons Learned about Searching for Precise Biomedical Scientific Information on COVID-19
This report describes the participation of two Danish universities,
University of Copenhagen and Aalborg University, in the international search
engine competition on COVID-19 (the 2020 TREC-COVID Challenge) organised by the
U.S. National Institute of Standards and Technology (NIST) and its Text
Retrieval Conference (TREC) division. The aim of the competition was to find
the best search engine strategy for retrieving precise biomedical scientific
information on COVID-19 from the largest, at that point in time, dataset of
curated scientific literature on COVID-19 -- the COVID-19 Open Research Dataset
(CORD-19). CORD-19 was the result of a call to action to the tech community by
the U.S. White House in March 2020, and was shortly thereafter posted on Kaggle
as an AI competition by the Allen Institute for AI, the Chan Zuckerberg
Initiative, Georgetown University's Center for Security and Emerging
Technology, Microsoft, and the National Library of Medicine at the US National
Institutes of Health. CORD-19 contained over 200,000 scholarly articles (of
which more than 100,000 were with full text) about COVID-19, SARS-CoV-2, and
related coronaviruses, gathered from curated biomedical sources. The TREC-COVID
challenge asked for the best way to (a) retrieve accurate and precise
scientific information, in response to some queries formulated by biomedical
experts, and (b) rank this information decreasingly by its relevance to the
query.
In this document, we describe the TREC-COVID competition setup, our
participation to it, and our resulting reflections and lessons learned about
the state-of-art technology when faced with the acute task of retrieving
precise scientific information from a rapidly growing corpus of literature, in
response to highly specialised queries, in the middle of a pandemic
Overview of CheckThat! 2020: Automatic Identification and Verification of Claims in Social Media
We present an overview of the third edition of the CheckThat! Lab at CLEF
2020. The lab featured five tasks in two different languages: English and
Arabic. The first four tasks compose the full pipeline of claim verification in
social media: Task 1 on check-worthiness estimation, Task 2 on retrieving
previously fact-checked claims, Task 3 on evidence retrieval, and Task 4 on
claim verification. The lab is completed with Task 5 on check-worthiness
estimation in political debates and speeches. A total of 67 teams registered to
participate in the lab (up from 47 at CLEF 2019), and 23 of them actually
submitted runs (compared to 14 at CLEF 2019). Most teams used deep neural
networks based on BERT, LSTMs, or CNNs, and achieved sizable improvements over
the baselines on all tasks. Here we describe the tasks setup, the evaluation
results, and a summary of the approaches used by the participants, and we
discuss some lessons learned. Last but not least, we release to the research
community all datasets from the lab as well as the evaluation scripts, which
should enable further research in the important tasks of check-worthiness
estimation and automatic claim verification.Comment: Check-Worthiness Estimation, Fact-Checking, Veracity, Evidence-based
Verification, Detecting Previously Fact-Checked Claims, Social Media
Verification, Computational Journalism, COVID-1
Overview of the CLEF-2022 CheckThat! Lab Task 1 on Identifying Relevant Claims in Tweets
We present an overview of CheckThat! lab 2022 Task 1, part of the 2022 Conference and Labs of the Evaluation Forum (CLEF). Task 1 asked to predict which posts in a Twitter stream are worth fact-checking, focusing on COVID-19 and politics in six languages: Arabic, Bulgarian, Dutch, English, Spanish, and Turkish. A total of 19 teams participated and most submissions managed to achieve sizable improvements over the baselines using Transformer-based models such as BERT and GPT-3. Across the four subtasks, approaches that targetted multiple languages (be it individually or in conjunction, in general obtained the best performance. We describe the dataset and the task setup, including the evaluation settings, and we give a brief overview of the participating systems. As usual in the CheckThat! lab, we release to the research community all datasets from the lab as well as the evaluation scripts, which should enable further research on finding relevant tweets that can help different stakeholders such as fact-checkers, journalists, and policymakers
Entity-Assisted Language Models for Identifying Check-worthy Sentences
We propose a new uniform framework for text classification and ranking that
can automate the process of identifying check-worthy sentences in political
debates and speech transcripts. Our framework combines the semantic analysis of
the sentences, with additional entity embeddings obtained through the
identified entities within the sentences. In particular, we analyse the
semantic meaning of each sentence using state-of-the-art neural language models
such as BERT, ALBERT, and RoBERTa, while embeddings for entities are obtained
from knowledge graph (KG) embedding models. Specifically, we instantiate our
framework using five different language models, entity embeddings obtained from
six different KG embedding models, as well as two combination methods leading
to several Entity-Assisted neural language models. We extensively evaluate the
effectiveness of our framework using two publicly available datasets from the
CLEF' 2019 & 2020 CheckThat! Labs. Our results show that the neural language
models significantly outperform traditional TF.IDF and LSTM methods. In
addition, we show that the ALBERT model is consistently the most effective
model among all the tested neural language models. Our entity embeddings
significantly outperform other existing approaches from the literature that are
based on similarity and relatedness scores between the entities in a sentence,
when used alongside a KG embedding.Comment: 22 pages, 15 tables, 3 figure