1 research outputs found
Credible, Unreliable or Leaked?: Evidence Verification for Enhanced Automated Fact-checking
Automated fact-checking (AFC) is garnering increasing attention by
researchers aiming to help fact-checkers combat the increasing spread of
misinformation online. While many existing AFC methods incorporate external
information from the Web to help examine the veracity of claims, they often
overlook the importance of verifying the source and quality of collected
"evidence". One overlooked challenge involves the reliance on "leaked
evidence", information gathered directly from fact-checking websites and used
to train AFC systems, resulting in an unrealistic setting for early
misinformation detection. Similarly, the inclusion of information from
unreliable sources can undermine the effectiveness of AFC systems. To address
these challenges, we present a comprehensive approach to evidence verification
and filtering. We create the "CREDible, Unreliable or LEaked" (CREDULE)
dataset, which consists of 91,632 articles classified as Credible, Unreliable
and Fact checked (Leaked). Additionally, we introduce the EVidence VERification
Network (EVVER-Net), trained on CREDULE to detect leaked and unreliable
evidence in both short and long texts. EVVER-Net can be used to filter evidence
collected from the Web, thus enhancing the robustness of end-to-end AFC
systems. We experiment with various language models and show that EVVER-Net can
demonstrate impressive performance of up to 91.5% and 94.4% accuracy, while
leveraging domain credibility scores along with short or long texts,
respectively. Finally, we assess the evidence provided by widely-used
fact-checking datasets including LIAR-PLUS, MOCHEG, FACTIFY, NewsCLIPpings+ and
VERITE, some of which exhibit concerning rates of leaked and unreliable
evidence