2 research outputs found
On the Importance of Delexicalization for Fact Verification
In this work we aim to understand and estimate the importance that a neural
network assigns to various aspects of the data while learning and making
predictions. Here we focus on the recognizing textual entailment (RTE) task and
its application to fact verification. In this context, the contributions of
this work are as follows. We investigate the attention weights a state of the
art RTE method assigns to input tokens in the RTE component of fact
verification systems, and confirm that most of the weight is assigned to POS
tags of nouns (e.g., NN, NNP etc.) or their phrases. To verify that these
lexicalized models transfer poorly, we implement a domain transfer experiment
where a RTE component is trained on the FEVER data, and tested on the Fake News
Challenge (FNC) dataset. As expected, even though this method achieves high
accuracy when evaluated in the same domain, the performance in the target
domain is poor, marginally above chance.To mitigate this dependence on
lexicalized information, we experiment with several strategies for masking out
names by replacing them with their semantic category, coupled with a unique
identifier to mark that the same or new entities are referenced between claim
and evidence. The results show that, while the performance on the FEVER dataset
remains at par with that of the model trained on lexicalized data, it improves
significantly when tested in the FNC dataset. Thus our experiments demonstrate
that our strategy is successful in mitigating the dependency on lexical
information.Comment: published in the proceedings at EMNLP201
Text Versus Paratext: Understanding Individualsâ Accuracy in Assessing Online Information
Fake news has emerged as a significant problem for society. Recent research has shown that shifting attention to accuracy improves the quality of content shared by individuals, thereby helping us mitigate the harmful effects of fake news. However, the parts of a news story that can influence individualsâ ability to discern the true state of information presented to them are understudied. We conducted an online experiment (N=408) to determine how different elements (text and paratext) of a news story influence individualsâ ability to detect the true state of the information presented. The participants were presented with the headline (control), main text, graphs/images, and sharing statistics of true and fake news stories and asked to evaluate the story's accuracy based on each of these elements separately. Our findings indicate that individuals were less accurate when identifying fake news from headlines, text, and graphs/images. When asked to evaluate the story based on sharing statistics, they were able to distinguish fake stories from real news with greater accuracy. Our findings also indicate that heuristics that apply to true news are ineffective for detecting the veracity of fake news