We present Claim-Dissector: a novel latent variable model for fact-checking
and analysis, which given a claim and a set of retrieved evidences jointly
learns to identify: (i) the relevant evidences to the given claim, (ii) the
veracity of the claim. We propose to disentangle the per-evidence relevance
probability and its contribution to the final veracity probability in an
interpretable way -- the final veracity probability is proportional to a linear
ensemble of per-evidence relevance probabilities. In this way, the individual
contributions of evidences towards the final predicted probability can be
identified. In per-evidence relevance probability, our model can further
distinguish whether each relevant evidence is supporting (S) or refuting (R)
the claim. This allows to quantify how much the S/R probability contributes to
the final verdict or to detect disagreeing evidence.
Despite its interpretable nature, our system achieves results competitive
with state-of-the-art on the FEVER dataset, as compared to typical two-stage
system pipelines, while using significantly fewer parameters. It also sets new
state-of-the-art on FAVIQ and RealFC datasets. Furthermore, our analysis shows
that our model can learn fine-grained relevance cues while using coarse-grained
supervision, and we demonstrate it in 2 ways. (i) We show that our model can
achieve competitive sentence recall while using only paragraph-level relevance
supervision. (ii) Traversing towards the finest granularity of relevance, we
show that our model is capable of identifying relevance at the token level. To
do this, we present a new benchmark TLR-FEVER focusing on token-level
interpretability -- humans annotate tokens in relevant evidences they
considered essential when making their judgment. Then we measure how similar
are these annotations to the tokens our model is focusing on.Comment: updated acknowledgemen