7 research outputs found
FACTIFY-5WQA: 5W Aspect-based Fact Verification through Question Answering
Automatic fact verification has received significant attention recently.
Contemporary automatic fact-checking systems focus on estimating truthfulness
using numerical scores which are not human-interpretable. A human fact-checker
generally follows several logical steps to verify a verisimilitude claim and
conclude whether its truthful or a mere masquerade. Popular fact-checking
websites follow a common structure for fact categorization such as half true,
half false, false, pants on fire, etc. Therefore, it is necessary to have an
aspect-based (delineating which part(s) are true and which are false)
explainable system that can assist human fact-checkers in asking relevant
questions related to a fact, which can then be validated separately to reach a
final verdict. In this paper, we propose a 5W framework (who, what, when,
where, and why) for question-answer-based fact explainability. To that end, we
present a semi-automatically generated dataset called FACTIFY-5WQA, which
consists of 391, 041 facts along with relevant 5W QAs - underscoring our major
contribution to this paper. A semantic role labeling system has been utilized
to locate 5Ws, which generates QA pairs for claims using a masked language
model. Finally, we report a baseline QA system to automatically locate those
answers from evidence documents, which can serve as a baseline for future
research in the field. Lastly, we propose a robust fact verification system
that takes paraphrased claims and automatically validates them. The dataset and
the baseline model are available at https: //github.com/ankuranii/acl-5W-QAComment: Accepted at ACL main conference 202
Findings of Factify 2: Multimodal Fake News Detection
With social media usage growing exponentially in the past few years, fake
news has also become extremely prevalent. The detrimental impact of fake news
emphasizes the need for research focused on automating the detection of false
information and verifying its accuracy. In this work, we present the outcome of
the Factify 2 shared task, which provides a multi-modal fact verification and
satire news dataset, as part of the DeFactify 2 workshop at AAAI'23. The data
calls for a comparison based approach to the task by pairing social media
claims with supporting documents, with both text and image, divided into 5
classes based on multi-modal relations. In the second iteration of this task we
had over 60 participants and 9 final test-set submissions. The best
performances came from the use of DeBERTa for text and Swinv2 and CLIP for
image. The highest F1 score averaged for all five classes was 81.82%.Comment: Defactify2 @AAAI 202
Factify 2: A Multimodal Fake News and Satire News Dataset
The internet gives the world an open platform to express their views and
share their stories. While this is very valuable, it makes fake news one of our
society's most pressing problems. Manual fact checking process is time
consuming, which makes it challenging to disprove misleading assertions before
they cause significant harm. This is he driving interest in automatic fact or
claim verification. Some of the existing datasets aim to support development of
automating fact-checking techniques, however, most of them are text based.
Multi-modal fact verification has received relatively scant attention. In this
paper, we provide a multi-modal fact-checking dataset called FACTIFY 2,
improving Factify 1 by using new data sources and adding satire articles.
Factify 2 has 50,000 new data instances. Similar to FACTIFY 1.0, we have three
broad categories - support, no-evidence, and refute, with sub-categories based
on the entailment of visual and textual data. We also provide a BERT and Vison
Transformer based baseline, which acheives 65% F1 score in the test set. The
baseline codes and the dataset will be made available at
https://github.com/surya1701/Factify-2.0.Comment: Defactify@AAAI202
Overview of Memotion 3: Sentiment and Emotion Analysis of Codemixed Hinglish Memes
Analyzing memes on the internet has emerged as a crucial endeavor due to the
impact this multi-modal form of content wields in shaping online discourse.
Memes have become a powerful tool for expressing emotions and sentiments,
possibly even spreading hate and misinformation, through humor and sarcasm. In
this paper, we present the overview of the Memotion 3 shared task, as part of
the DeFactify 2 workshop at AAAI-23. The task released an annotated dataset of
Hindi-English code-mixed memes based on their Sentiment (Task A), Emotion (Task
B), and Emotion intensity (Task C). Each of these is defined as an individual
task and the participants are ranked separately for each task. Over 50 teams
registered for the shared task and 5 made final submissions to the test set of
the Memotion 3 dataset. CLIP, BERT modifications, ViT etc. were the most
popular models among the participants along with approaches such as
Student-Teacher model, Fusion, and Ensembling. The best final F1 score for Task
A is 34.41, Task B is 79.77 and Task C is 59.82.Comment: Defactify2 @AAAI 202
FACTIFY3M: A Benchmark for Multimodal Fact Verification with Explainability through 5W Question-Answering
Combating disinformation is one of the burning societal crises -- about 67%
of the American population believes that disinformation produces a lot of
uncertainty, and 10% of them knowingly propagate disinformation. Evidence shows
that disinformation can manipulate democratic processes and public opinion,
causing disruption in the share market, panic and anxiety in society, and even
death during crises. Therefore, disinformation should be identified promptly
and, if possible, mitigated. With approximately 3.2 billion images and 720,000
hours of video shared online daily on social media platforms, scalable
detection of multimodal disinformation requires efficient fact verification.
Despite progress in automatic text-based fact verification (e.g., FEVER, LIAR),
the research community lacks substantial effort in multimodal fact
verification. To address this gap, we introduce FACTIFY 3M, a dataset of 3
million samples that pushes the boundaries of the domain of fact verification
via a multimodal fake news dataset, in addition to offering explainability
through the concept of 5W question-answering. Salient features of the dataset
include: (i) textual claims, (ii) ChatGPT-generated paraphrased claims, (iii)
associated images, (iv) stable diffusion-generated additional images (i.e.,
visual paraphrases), (v) pixel-level image heatmap to foster image-text
explainability of the claim, (vi) 5W QA pairs, and (vii) adversarial fake news
stories.Comment: arXiv admin note: text overlap with arXiv:2305.0432