International audienceEnglish. In the latest years, a number of real world applications have underlined the need to move from Textual Entailment (TE) pairs to TE graphs where pairs are no more independent. Moving from single pairs to a graph has the advantage of providing an overall view of the issue discussed in the text, but this may lead to possible inconsistencies due to the combination of the TE pairs into a unique graph. In this paper, we adopt argumentation theory to support human annotators in detecting the possible sources of inconsistencies. Italiano. Negli ultimi anni, in svari-ate applicazioni sta sorgendo la necessit a di passare da coppie di Textual Entail-ment (TE) a grafi di TE, in cui le cop-pie sono interconnesse. Il vantaggio dei grafi di TE e di fornire una visione glob-ale del soggetto di cui si sta discutendo nel testo. Allo stesso tempo, questopù o gener-are inconsistenze dovute all'integrazione dipì u coppie di TE in un unico grafo. In questo articolo, ci basiamo sulla teo-ria dell'argomentazione per supportare gli annotatori nell'individuare le possibili fonti di inconsistenze

Cabrio, Elena

Villata, Serena

In the latest years, a number of real world applications have underlined the need to move from Textual Entailment (TE) pairs to TE graphs where pairs are no more independent. Moving from single pairs to a graph has the advantage of providing an overall view of the issue discussed in the text, but this may lead to possible inconsistencies due to the combination of the TE pairs into a unique graph. In this paper, we adopt argumentation theory to support human annotators in detecting the possible sources of inconsistencies.Negli ultimi anni, in svariate applicazioni sta sorgendo la necessità di passare da coppie di Textual Entailment (TE) a grafi di TE, in cui le coppie sono interconnesse. Il vantaggio dei grafi di TE e di fornire una visione globale del soggetto di cui si sta discutendo nel testo. Allo stesso tempo, questo puo generare inconsistenze dovute all’integrazione di piu coppie di TE in un unico grafo. In questo articolo, ci basiamo sulla teoria dell’argomentazione per supportare gli annotatori nell’individuare le possibili fonti di inconsistenze

OpenEdition

Inconsistencies Detection in Bipolar Entailment Graphs

HAL-UNICE

INRIA a CCSD electronic archive server

HAL Id: hal-01236707https://hal.inria.fr/hal-01236707Submitted on 2 Dec 2015HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.Distributed under a Creative Commons Attribution| 4.0 International LicenseInconsistencies Detection in Bipolar Entailment GraphsElena Cabrio, Serena VillataTo cite this version:Elena Cabrio, Serena Villata. Inconsistencies Detection in Bipolar Entailment Graphs. Second ItalianConference on Computational Linguistics - CLiC-it 2015, Dec 2015, Trento, Italy. ￿hal-01236707￿Inconsistencies Detection in Bipolar Entailment GraphsElena Cabrio1 and Serena Villata22 CNRS, 1,2University of Nice Sophia Antipolis, Franceelena.cabrio@unice.fr; serena.villata@cnrs.frAbstractEnglish. In the latest years, a numberof real world applications have underlinedthe need to move from Textual Entailment(TE) pairs to TE graphs where pairs areno more independent. Moving from sin-gle pairs to a graph has the advantage ofproviding an overall view of the issue dis-cussed in the text, but this may lead to pos-sible inconsistencies due to the combina-tion of the TE pairs into a unique graph. Inthis paper, we adopt argumentation theoryto support human annotators in detectingthe possible sources of inconsistencies.Italiano. Negli ultimi anni, in svari-ate applicazioni sta sorgendo la necessitàdi passare da coppie di Textual Entail-ment (TE) a grafi di TE, in cui le cop-pie sono interconnesse. Il vantaggio deigrafi di TE è di fornire una visione glob-ale del soggetto di cui si sta discutendo neltesto. Allo stesso tempo, questo può gener-are inconsistenze dovute all’integrazionedi più coppie di TE in un unico grafo.In questo articolo, ci basiamo sulla teo-ria dell’argomentazione per supportaregli annotatori nell’individuare le possibilifonti di inconsistenze.1 IntroductionA Textual Entailment (TE) system (Dagan et al.,2009) automatically assigns to independent pairsof two textual fragments either an entailment ora contradiction relation. However, in some realworld scenarios like analyzing costumer reviewsabout a service or product, these pairs cannot beconsidered as independent. For instance, all the re-views about a certain service need to be collectedinto a single graph, to understand the overall prob-lems/merits of the service. The combination of TEpairs into a unique graph may generate inconsis-tencies due to the wrong relation assignment bythe TE system, which could not have been iden-tified if TE pairs were considered independently.The detection of such inconsistencies is usuallyleft to human annotators, which later correct them.The need of processing such graphs to support an-notators is therefore of crucial importance, partic-ularly when dealing with big amounts of data. Ourresearch question is How to support annotators indetecting inconsistencies in TE graphs?The term entailment graph has been introducedby (Berant et al., 2010) as a structure to modelentailment relations between propositional tem-plates. Differently, in this paper we consider bipo-lar entailment graphs (BEGs), where two kinds ofedges are considered, i.e., entailment and contra-diction, to reason over the graph consistency.We answer the research question by adoptingabstract argumentation theory (Dung, 1995), areasoning framework used to detect and solveinconsistencies in the so-called argumentationgraphs, where nodes are called arguments, andedges represent a conflict relation. Argumentationsemantics allows to compute consistent sets of ar-guments, given the conflicts among them.We define the BEGincs (BEG-Inconsistencies)framework, which translates a BEG into an argu-mentation graph. It then provides to the annota-tors sets of arguments, following argumentationsemantics, that are supposed to be consistent. Ifit is not the case, the TE system wrongly assignedsome relations. Moving from single pairs to anoverall graph allows for the detection of inconsis-tencies otherwise undiscovered. BEGincs does notidentify the precise relation causing the inconsis-tency, but providing annotators with the consistentarguments sets, they are supported in narrowingthe causes of inconsistency.2 BEGincs frameworkTE is a directional relation between two textualfragments. In various real world scenarios, thesepairs cannot be considered as independent, andthey need to be collected into a single graph. Wedefine therefore a new framework involving en-tailment graphs, where pairs of textual fragmentsconnected by semantic relations are also part of agraph that provides an overall view of the state-ments’ interactions (bipolar entailment graphs).Definition 1. A bipolar entailment graph is a tu-ple BEG = 〈T,E,C〉 where T is a set of textfragments, E ⊆ T × T is an entailment relationbetween text fragments, and C ⊆ T × T is a con-tradiction relation between text fragments.This opens new challenges for TE, that originallyconsiders the pairs as “self-contained” (i.e., themeaning of one text has to be derived from themeaning of the other). One challenge consists inchecking BEGs to identify possible inconsisten-cies due to wrong relation assignments by the TEsystem. Figure 1 shows the architecture of the BE-Gincs framework to support human annotators indetecting inconsistencies in TE graphs.RTEENGINEBEG-AFINCONSISTENCIESDETECTIONFRAMEWORK1 2BEGincs FRAMEWORKAE DCBFigure 1: The BEGincs framework architecture.Annotators provide the dataset to be checked asinput of the BEGincs framework, which consistsof two main modules: (1) a TE module, takes asinput the dataset of text fragments, and returns thepairs annotated with the entailment or contradic-tion relations; and (2) a BEG-AF InconsistenciesDetection module, which translates the receivedBEGs into an argumentation framework such thatargumentation semantics can be applied to retrieveconsistent sets of arguments. The BEGincs frame-work returns through a user interface the startingBEGs highlighted with the consistent sets of textfragments. Checking them, annotators are able todetect errors in the annotation produced by the TEmodule (they will find inconsistent arguments inthe returned sets), and correct the erroneous pairs.2.1 Argumentation theoryAn abstract argumentation framework(AF) (Dung, 1995) represents conflicts amongelements called arguments. It is based on abinary attack relation among them, whose roleis determined only by their relation with theother arguments. An AF encodes, through theattack relation, the existing conflicts within a setof arguments. It identifies then the conflict out-comes, i.e. which arguments should be accepted(“they survive the conflict”) and which argumentsshould be rejected, according to some reason-able criterion. (Dung, 1995) presents severalacceptability semantics that produce zero, one,or several consistent sets of accepted arguments.Such set of accepted arguments does not containan argument conflicting with another argument inthe set (conflict free). Following from this notion,an admissible set of arguments is required to beboth internally coherent (conflict-free) and ableto defend its elements. In BEGincs, we adoptadmissibility based semantics. Roughly, an argu-ment is accepted if all the arguments attacking itare rejected, and it is rejected if there is at leastan argument attacking it which is accepted. Thesets of accepted arguments computed using anacceptability semantics are called extensions, andthe addition of another argument from outside theset will make it inconsistent.2.2 Inconsistencies detectionTo reuse abstract argumentation results and se-mantics for inconsistencies detection, we need torepresent both the entailment and the contradictionrelations of the bipolar entailment graph under theform of attacks between abstract arguments in anargumentation graph (Definition 2).Definition 2. A BEG-based argumentation frame-work is a tuple 〈A,⇒,⇔〉 where A is a set of textfragments called arguments,⇒ is a binary entail-ment relation on A (⇒ ⊆ A × A), and ⇔ is abinary contradiction relation on A (⇔⊆ A×A).The set of arguments is {a, b, . . . ∈ A}.BEG-AFs’ consistent sets of arguments containthe text fragments that do not conflict with otherfragments in the set (they are coherent). BEGincsuses the consistent sets of arguments computedfollowing admissibility based argumentation se-mantics to support annotators in detecting incon-sistencies. We need then to define the semanticsof the entailment and contradiction relations in theBEG-based argumentation framework (i.e. the be-havior these relations have to satisfy in terms ofconflict, since the only relation between argumentsin abstract argumentation is the conflict relation).Example 1.T1: Natural gas vehicles run on natural gas,so emit significant amounts of greenhouse gasesinto the atmosphere, albeit smaller amounts thangasoline-fueled cars. To combat global warming,we should be focusing our energies and invest-ments solely on 0-emission electric vehicles.H: On the surface, natural gas cars seem alright,but the topic becomes a bit different when they arecompeting against zero emission alternatives (e.g.electric cars).In Example 1, the text (T1) entails the hypoth-esis (H), i.e., T1 ⇒ H . Entailment is a direc-tional relation (Dagan et al., 2009), that holds ifthe meaning of H can be inferred from the mean-ing of T , as interpreted by a typical languageuser. In the pair, T is more specific than H (i.e.,the more specific argument entails the more gen-eral one). In the argumentation setting, we haveto reason over this feature to identify which con-straints it poses in terms of conflicts among thetext fragments. In particular, the following con-straints emerge from the entailment relation: as-suming T entails H holds, then (i) if there is atext fragment T1 which contradicts H (negativeTE) then T1 contradicts also T (T ≡ T1 doesnot entail H ≡ T ), and (ii) if there is a text frag-ment T2 which contradicts T then T2 does not nec-essary contradict H too. These two constraintshold when a TE pair is inserted into an entailmentgraph. As a consequence, from the arguments’acceptance viewpoint: given that T ⇒ H , everytime argument H is rejected, argument T is re-jected too. We model the entailment relation suchthat, given that T entails H , T is accepted only ifH is accepted too (Definit. 3)1.Definition 3. Given a BEG-based argumentationframework 〈A,⇒,⇔〉, a translated BEG-basedargumentation framework (BEG-AF) is a tuple〈A, 7−→〉 such that the set of arguments A is{a, b, . . . ∈ A} ∪ {Xa,b, Ya,b, Ea,b | a, b ∈ A},where Xa,b, Ya,b are the dummy arguments corre-sponding to the contradiction relation and Ea,b isthe dummy argument corresponding to the entail-ment relation, and 7−→ is a binary conflict relation1See (Cabrio and Villata, 2013) for a comparison of theentailment wrt the support relation (Boella et al., 2010).over A such that: b 7−→ Ea,b 7−→ a iff a⇒ b.We have now to define the semantics of the con-tradiction relation (i.e., negative TE) in BEGs, seeExample 2. (Marneffe et al., 2008) claims thatcontradiction occurs when two sentences i) are ex-tremely unlikely to be true simultaneously, and ii)involve the same event. Starting from these con-siderations, the following constraint holds for thecontradiction pairs: T and H conflict with eachother (i.e. it is not possible to have both in a co-herent and consistent set of arguments).Example 2.T2: Natural gas is the cleanest transportation fuelavailable today. If we want to immediately beginthe process of significantly reducing greenhousegas emissions, natural gas can help now. Otheralternatives cannot be pursued as quickly.H: On the surface, natural gas cars seem alright,but the topic becomes a bit different when they arecompeting against zero emission alternatives (e.g.electric cars).Definition 4 models contradiction in BEG-AFs.The attack in (Dung, 1995) is directed from an ar-gument to another argument while our contradic-tion leads to a cycle of attacks.Definition 4. Given a BEG-based argumenta-tion framework 〈A,⇒,⇔〉, a BEG-AF is a tuple〈A, 7−→〉 such that A is the set of arguments, and7−→ is a binary conflict relation over A such that:a 7−→ Xa,b 7−→ Ya,b 7−→ b, andb 7−→ Xb,a 7−→ Yb,a 7−→ a, iff a⇔ b.Figure 2 summarizes the translation procedure,which is the core of our framework. We start witha BEG consisting of three text fragments (i.e., ar-guments A, B, C) from Ex. 1 and 2, where T1 isA, T2 is B, and H is C. The BEG is then translatedinto a BEG-AF where dummy arguments are in-troduced to express the semantics of the relationsof entailment and contradiction, e.g., dummy ar-gument EA,C represents the relation A entails C inthe BEG-AF. The only relation allowed in a BEG-AF is the conflict relation 7−→. Therefore we havethat a BEG-AF is a standard abstract AF, and wecan apply admissibility based argumentation se-mantics to retrieve consistent sets of arguments.Acceptability semantics return the extension of theBEG-AF (i.e., the black nodes in Fig. 2), where ar-guments C,A are accepted, and dummy argumentsare filtered out from the set of accepted ones.We prove now that our BEG-AF actually satis-fies the semantics of the entailment relation.ACBBX(B,C) Y(B,C)CY(C,B) X(C,B)AE(A,C)BEGBEG-AFFigure 2: Translation from a BEG to a BEG-AF.Proposition 1 (Semantics of entailment). Given aBEG-AF, if it holds that T ⇒ H and text fragmentT is accepted, then fragment H is accepted too.Proof. We prove the contrapositive. If it holds thatT ⇒ H and text fragment H is not accepted,then text fragment T is not accepted. Assume thatT ⇒ H and assume that argument H is not ac-cepted, then dummy argument ET,H is accepted.Consequently, T is not accepted, i.e., rejected.We need to add two nodes, i.e., dummy argu-ments Xa,b and Ya,b, to represent a contradictionwhile we only need one node, i.e., dummy argu-ment Ea,b, to represent entailment, since preserv-ing the semantics of a contradiction holding be-tween two text fragments means that the two textfragments cannot be together in a consistent setof arguments. To avoid the two being both ac-cepted, we need to introduce two dummy argu-ments so that: a (accepted) 7−→ Xa,b (rejected),Xa,b 7−→ Ya,b (accepted), and Ya,b 7−→ b (re-jected). In this way, if a is accepted then b is re-jected, and viceversa. A unique dummy argumentbetween a and b would not ensure such behavior.Existing works combine NLP and argumenta-tion theory, e.g. (Chesñevar and Maguitman, 2004;Carenini and Moore, 2006; Wyner and van Engers,2010; Feng and Hirst, 2011) with different pur-poses. However, only our previous work (Cabrioand Villata, 2012) combines TE with AF, but hereour goal is to introduce a framework for inconsis-tencies detection in TE annotations.3 Experimental settingData set. We added 60 pairs to the Debatepediadataset 2 (extracted from a sample of Debatepedia3debates (Cabrio and Villata, 2012)), resulting in160 pairs as training set, and 100 pairs as test set(balanced wrt to entailment/contradiction).2The only available dataset of T-H pairs combined intobipolar entailment graphs.3http://idebate.org/Evaluation. First step: we assess the perfor-mances of the TE system to correctly assign theTE relations to the pairs of arguments in thedataset. Second step: we evaluate how much suchperformances impact on the flattening of the BEG-AF, i.e., how much a wrong assignment of a rela-tion to a pair of arguments is propagated in the AF.It is actually to detect such wrong assignments thatthe BEGincs framework has been conceived.To recognize TE, we tested several algorithmsfrom the EOP4, i.e. BIUTEE (Stern and Da-gan, 2011), TIE5 and EDITS (Kouylekov and Ne-gri, 2010). BIUTEE obtained the best resultson Debatepedia (configuration exploiting all avail-able knowledge resources): Acc:0.71, Rec:0.94,Pr:0.66, F-meas:0.78. As baseline we use atoken-based version of the Levenshtein distancealgorithm, i.e. EditDistanceEDA in the EOP(Acc:0.58, Rec:0.61, Pr:0.59, F-meas:0.59).Then, we consider the impact of the best TEconfiguration on the arguments acceptability. Weuse admissibility-based semantics to identify theaccepted arguments both on i) the goldstandardentailment graphs of Debatepedia topics, andii) on the graphs generated using the relationsassigned by BIUTEE. On the 10 Debatepediagraphs, BEGincs avg pr:0.68, avg rec:0.91, F-meas:0.77. BIUTEE mistakes in relation assign-ment propagate in the AF, but results are promis-ing. The incons. detection module takes ∼1 sec.to analyze a BEG of 100 nodes and 150 relations.4 Concluding remarksWe have presented BEGincs, a new formal frame-work that, translating a BEG into an argumenta-tion graph, returns inconsistent set of arguments,if a wrong relation assignment by the TE systemoccurred. These inconsistent arguments sets arethen used by annotators to detect the presence ofa wrong assignment, and if so, to narrow the setof possibly erroneous relations. If no mistakes areproduced in relation assignment, by definition BE-Gincs semantics return consistent arguments sets.Assuming that in several real world scenariosTE pairs are interconnected, we ask to the NLPcommunity to contribute in the effort of buildingsuitable resources. In BEGincs, we plan to verifyand ensure transitivity of BEGs.4http://bit.ly/ExcitementOpenPlatform5http://bit.ly/MaxEntClassificationEDAReferencesJ. Berant, I. Dagan, and J. Goldberger. 2010. Globallearning of focused entailment graphs. In ACL,pages 1220–1229.G. Boella, D. M. Gabbay, L. W. N. van der Torre,and S. Villata. 2010. Support in abstract argu-mentation. In P. Baroni, F. Cerutti, M. Giacomin,and G. R. Simari, editors, COMMA, volume 216 ofFrontiers in Artificial Intelligence and Applications,pages 111–122. IOS Press.E. Cabrio and S. Villata. 2012. Natural language ar-guments: A combined approach. In Procs of ECAI,Frontiers in Artificial Intelligence and Applications242, pages 205–210.E. Cabrio and S. Villata. 2013. A natural languagebipolar argumentation approach to support users inonline debate interactions;. Argument & Computa-tion, 4(3):209–230.G. Carenini and J. D. Moore. 2006. Generatingand evaluating evaluative arguments. Artif. Intell.,170(11):925–952.C. I. Chesñevar and A.G. Maguitman. 2004. An ar-gumentative approach to assessing natural languageusage based on the web corpus. In Procs of ECAI,pages 581–585.I. Dagan, B. Dolan, B. Magnini, and D. Roth. 2009.Recognizing textual entailment: Rational, evalua-tion and approaches. Natural Language Engineer-ing (JNLE), 15(04):i–xvii.P.M. Dung. 1995. On the acceptability of argumentsand its fundamental role in nonmonotonic reason-ing, logic programming and n-person games. Artif.Intell., 77(2):321–358.V. Wei Feng and G. Hirst. 2011. Classifying argu-ments by scheme. In Procs of ACL-2012, pages987–996.M. Kouylekov and M. Negri. 2010. An open-sourcepackage for recognizing textual entailment. In Procsof ACL 2010 System Demonstrations, pages 42–47.M.C. De Marneffe, A.N. Rafferty, and C.D. Manning.2008. Finding contradictions in text. In Procs ofACL.A. Stern and I. Dagan. 2011. A confidence model forsyntactically-motivated entailment proofs. In Pro-ceedings of RANLP 2011.A. Wyner and T. van Engers. 2010. A framework forenriched, controlled on-line discussion forums for e-government policy-making. In Procs of eGov 2010.

https://hal.inria.fr/hal-01236707/file/clic2015_CR.pdf

Inconsistencies Detection in Bipolar Entailment Graphs

Abstract

Similar works

Full text

Available Versions

OpenEdition

HAL-UNICE

INRIA a CCSD electronic archive server