This paper reports the description of the developed system and the results obtained in the participation of the UNED in the Second Recognizing Textual Entailment (RTE) Challenge. New techniques and tools have been added: enriched queries to WordNet, detection of numeric expressions and their entailment, and Support Vector Machine classi cation (SVM) are the more relevant. The accuracy performed is slightly higher than the one from the previous edition system

Herrera de la Cruz, Jesús

Peñas, Anselmo

Rodrigo, Álvaro

Verdejo, Felisa

This paper reports the description of the developed system and the results obtained in the participation of the UNED in the Second Recognizing Textual Entailment (RTE) Challenge. New techniques and tools have been added: enriched queries to WordNet, detection of numeric expressions and their entailment, and Support Vector Machine classi cation (SVM) are the more relevant. The accuracy performed is slightly higher than the one from the previous edition system.Depto. de Sistemas Informáticos y ComputaciónFac. de InformáticaTRUEinpres

Docta Complutense

UNED at PASCAL RTE-2 ChallengeJesús Herrera, Anselmo Peñas, Álvaro Rodrigo, Felisa VerdejoDepartmento de Lenguajes y Sistemas InformáticosUniversidad Nacional de Educación a DistanciaMadrid, Spain{jesus.herrera, anselmo, alvarory, felisa}@lsi.uned.esAbstractThis paper reports the description of thedeveloped system and the results obtainedin the participation of the UNED1 in theSecond Recognizing Textual Entailment(RTE) Challenge. New techniques andtools have been added: enriched queriesto WordNet, detection of numeric expres-sions and their entailment, and SupportVector Machine classification (SVM) arethe more relevant. The accuracy per-formed is slightly higher than the one fromthe previous edition system.1 IntroductionThe system presented to the Second RecognizingTextual Entailment Challenge is based on the onepresented to the First RTE Challenge (Herrera et al.,2005). The core of this latter was basically kept, butenhanced by means of several subsystems in order tostudy the efficiency of other not previously appliedtechniques that seemed promising for RTE.In short, the techniques involved in this new sys-tem are the following:• Dependency analysis of texts and hypothesises.• Lexical entailment between dependency treenodes using WordNet. The subsystem consult-ing WordNet was enriched with respect to theone presented to the First RTE Challenge.1Spanish Distance Learning University.• Mapping between dependency trees, which isthe one defined for the previous system (Her-rera et al., 2005).• Detection of numeric expressions. A newmodule, which detects entailment between nu-meric expressions of the texts and the hypoth-esises, was implemented. For this detection,the train and test corpora were automaticallytagged (cardinals, dates and named entities) bythe López-Ostenero’s system (Peinado et al.,2005).• Support Vector Machine classification in orderto determine the final decision about textual en-tailment between pairs of text and hypothesis,following previous ideas from successful worksin Natural Language Processing using machinelearning applications (Joachims, 1998).2 System DescriptionThe proposed system is based on surface techniquesof lexical and syntactic analysis, complemented withqueries to WordNet as an external source of knowl-edge. It works in a non-specific way, not givingany kind of special treatment for the different set-tings considered in the RTE Challenge (InformationRetrieval, Multi-document summarization, QuestionAnswering and Information Extraction).The system accepts pairs of text snippets (text andhypothesis) at the input and gives a boolean value atthe output: YES if the text entails the hypothesis andNO otherwise. This value is obtained by the appli-cation of the learned model by a SVM classifier.Figure 1: System’s architecture.System’s components, whose graphic representa-tion is shown in figure 1, are the following:2.1 Linguistic processingA dependency parser, based on Lin’s Minipar (Lin,1998), which normalizes data from the corpus oftext and hypothesis pairs and accomplishes the de-pendency analysis, generating a dependency tree forevery text and hypothesis.A named entities recognizer, implemented byLópez-Ostenero (Peinado et al., 2005), has beenused to normalize numeric expressions and namedentities.2.2 Lexical entailmentA WordNet-based entailment module – which takesthe information given by the parser and returns thehypothesis’ nodes that are entailed by the text (Her-rera et al., 2005) – uses WordNet in order to findsynonymy, similarity, hyponymy, WordNet’s entail-ment and negation relations between pairs of lexicalunits, as described in (Herrera et al., 2005). For thecurrent edition some features have been added to thelexical entailment module:• Search of entailment paths. It has been stud-ied whether the strategy for searching the en-tailment paths affects or not the results. Twostrategies have been tested: depth search andbreadth search. In addition, the length ofthe path has been used as a final criteria fordeciding the entailment between words. Al-though behaviour is slightly different, the ef-fect over results in the exercise is no significant.However, the breadth strategy is significativelyslower.• WordNet relations. Synonymy, hyponymy, verbentailment and antonymy have been used asin the last edition. In addition, part meronym(e.g. Italy entails Europe), and adjective / ad-verb pertainym (e.g. Italian entails Italy) havebeen added in the search of entailment pathsbetween the lemmas of the text and the ones ofthe hypothesis.Table 1: Entailment between numeric expressionsText HypothesisRecognition 17 million citizens more than 15 million peopleNormalization lower bound: 17, 000, 000 lower bound: 15, 000, 000upper bound: 17, 000, 000 upper bound: infiniteunit: citizen unit: personEntailment TRUE if 15, 000, 000 ≤ 17, 000, 000 and infinite ≥ 17, 000, 000 and citizen entails person• Entailment between phrases / multiwords. Lev-ensthein distance has been used for an approxi-mate matching between multiwords only if theone related to the hypothesis is present in Word-Net. A new and simple entailment relationbetween phrases has been defined assumingthe compositional meaning of phrases. Thus,a phrase is expected to entail all its compo-nents. This entailment relation can‘t be usedover Named Entities since they haven’t a com-positional meaning.• Entailment between numeric expressions. Nu-meric expressions from the corpus are detectedby means of an entities recognizer; they arenormalized after the recognition, and the unitsaffected (e.g. kilometers, years, etcetera) areconsidered for the detection of an entailmentrelation between these expressions. Thus, anumeric expression N1 entails a numeric ex-pression N2 if the range associated to N2 en-closes the range of N1 and the unit of N1 entailsthe unit of N2. When a numeric expression inthe hypothesis is not entailed by one or morenumeric expressions in the text, then the sys-tem responses that there is not entailment be-tween numeric expressions in the pair. An ex-ample is shown in table 1. The experiment infigure 2 shows the accuracy obtained over thedevelopment corpus when considering: coin-cidence between lemmas (LEM), WordNet re-lations (WN), and entailment between numericexpressions (NUMFAIL). For every percentageof overlap between the text and the hypothesis,the accuracy obtained is higher when addingNUMFAIL to the set of considered features; andthe lower is the overlap the higher is this accu-racy improvement. Thus, NUMFAIL is an in-teresting feature to decide if there is entailmentbetween a text and a hypothesis when a loweroverlap exists.The named entities (NE) entailment module, asdescribed in section 5, did not contribute to the runssubmitted.2.3 Sentence level matchingA tree matching module, which searches for match-ing branches into the hypothesises’ dependencytrees. These kind of branches are the ones whose allnodes are lexically entailed, as described in (Herreraet al., 2005).A plain text matching module that calculates thepercentage of lemmas from the hypothesis entailedby lemmas from the text, according to section 2.2.2.4 Entailment decisionA SVM classifier, from Yet Another Learning Envi-ronment Yale 3.0 (Fischer et al., 2005), which wasapplied in order to train a model from the develop-ment corpus given by the organization and to apply itto the test corpus. The model was trained by meansof a set of features obtained from the other mod-ules of the system; these ones, for every pair <text,hypothesis>, are the following:1. Percentage of nodes of the hypothesis’ de-pendency tree pertaining to matching branches(Herrera et al., 2005) considering, respectively:• Lexical entailment between the words ofthe snippets involved, without consultingWordNet.• Lexical entailment between the lemmasof the snippets involved, consulting Word-Net.2. Percentage of lemmas of the hypothesis en-tailed by lemmas of the text, considering thelexical entailment relations described in section2.2.Figure 2: Effect of the numerical entailment restriction over the development corpus3. Percentage of words of the hypothesis in thetext (treated as bags of words).4. Percentage of lemmas of the hypothesis in thetext (treated as bags of lemmas).5. Existence or absence of any numeric expres-sion within the hypothesis.6. Existence or absence of any numeric expres-sion within the text.7. Existence or absence of entailment between nu-meric expressions of the text and the hypothe-sis, as described in section 2.2.3 Runs SubmittedTwo runs were submitted to the Second RTE Chal-lenge.Run 1 was obtained using only the features 2 and7 of section 2.4.Run 2 was produced by the system described insection 2. The SVM was trained with the featuresenumerated in section 2.4, obtained from the devel-opment corpus provided by the organizers. Thus, themodel was applied to the features from the test cor-pus in order to obtain a prediction for the existenceor absence of textual entailment for every pair<text,hypothesis>.4 PerformanceTwo evaluation measures were applied to the partic-ipating systems: accuracy (Dagan et al. , 2005), asthe main measure, and average precision (Voorheesand Harman, 1999), as the secondary measure. Ac-curacy was computed for all the runs submitted, butaverage precision only for the runs giving the resultsranked according to their entailment confidence; thisranking was not mandatory.One of the two runs submitted was ranked and,then, average precision was computed for it. Theresults obtained over the test corpus are shown intables 2 and 3.The accuracy of the system is not homogeneousover all the pairs and depends on the different appli-Table 2: Results for run 1Accuracy Average PrecisionIE 49.00% 47.74%IR 64.50% 69.14%QA 56.50% 50.24%SUM 69.00% 79.05%Overall 59.75% 56.63%Table 3: Results for run 2AccuracyIE 52.00%IR 57.00%QA 52.00%SUM 74.50%Overall 58.87%cation settings proposed by the organizers: Informa-tion Retrieval (IR), Multi-document Summarization(SUM), Question Answering (QA) and InformationExtraction (IE). The overall accuracy shown by thecurrent system is basically due to the contributionof the Multi-document Summarization setting. Thissetting is characterized by sentence pairs with highlexical overlap, and the system shows its better ac-curacy for the subset of pairs pertaining to this kindof setting, for which reaches 74.50% accuracy.4.1 Performance comparisonThough the overall accuracy is better than the oneobtained in the First RTE Challenge, the improve-ment has not been very significant: only 3.38percentage points between the best performancesreached in every edition. Considering a combina-tion of the two systems used to obtain the runs sub-mitted to de Second RTE Challenge, in which theIR and QA pairs were treated by the system produc-ing run 1 and the IE and SUM pairs were treatedby the system producing run 2, the best performancecould be given. In such a case, the overall accuracywill be 61.88%; thus, the performance improvementwith respect to the previous edition of the Challengewill be of 5.51 percentage points.During the development of the system, some ex-periments were accomplished in order to computeits accuracy. Training the SVM with a half of thedevelopment corpus and applying the model to theother half, and vice-versa, the overall accuracy ob-tained ranged between 63% and 64%. It overcomesin more than 4 percentage points the best accuracyobtained by the two runs submitted. Since the de-velopment corpus and the test corpus are similar, itcan be concluded that the accuracy of the system isquite variable depending on the concrete samples ofthe corpus. Then, it is not easy to affirm that the cur-rent system is clearly better than the previous one.In the first edition of the RTE Challenge theMulti-document Summarization setting was not pro-posed but a similar one called Comparable Docu-ments (Dagan et al. , 2005), characterized by ahigh lexical overlap between texts and hypothesises,too. For Comparable Documents, the system pre-sented to the previous edition of the RTE Challengereached its best accuracy, with a 79.33%. The ac-curacy obtained for the settings with a high lexicaloverlap went slightly down from the first to the sec-ond edition. Because of the settings are not exactlythe same, no definitely conclusions can be stated, butit is clear that both systems are significantly good atrecognizing textual entailment when the pairs showa high lexical overlap.5 New Experiments after Second RTEChallenge ParticipationAfter submitting the results to the Second RTE Chal-lenge, the development of the system continued andsome experiments were accomplished.5.1 New features based on named entitiesA new module for recognizing entailment betweennamed entities (see figure 1) was implemented inorder to study its usefulness for RTE. It works ina similar way to the numerical entailment module.The features computed were the following: a) ex-istence or absence of any named entity within thehypothesis; b) existence or absence of any namedentity within the text; c) existence or absence of en-tailment between named entities of the text and thehypothesis; it is said that there is entailment betweennamed entities of the text and its correspondant hy-pothesis if, in case of existence of named entities inthe hypothesis, all them are entailed by one ore morenamed entities from the text.The results obtained after executing the wholesystem described in figure 1 – over the test corpus– are shown in table 4.Table 4: Results considering named entitiesAccuracyIE 51.00%IR 63.50%QA 53.50%SUM 74.00%Overall 60.50%The overall accuracy of this system is slightly bet-ter than the ones obtained by the other two systems.Thus, named entities seem to be an interesting fieldof study in order to improve RTE.6 Conclusions and Future WorkThe system presented to the First RTE Challenge hasbeen improved in order to put in the same level theweight of the analysis due to the overlap betweendependency trees – which represented the only de-cision item in the previous system – and the weightof other kinds of analysis, such as bag of words, bagof lemmas, numerical entailment, etcetera. The rele-vance of every feature computed for the pairs <text,hypothesis> has been determined automatically bya SVM algorithm (training with the developmentcorpus) which is the responsible for the predictionof existence/absence of entailment between the pairsof the test corpus.Despite the complexity of the developed systemis quite higher than the complexity of the previousone, the obtained accuracy is slightly better. It seemsnot easy to determine the way to obtain a higher ac-curacy for every application setting. With the cur-rently experimented techniques and tools, the resultsobtained for settings with a high lexical overlap aresignificantly higher than the others, which are quitesimilar among themselves. It suggest that it shouldbe stimulated the development of setting-orientedsystems, aiming to increase the performance of RTEsystems focusing on only one or a few settings.Thus, in the medium term, useful RTE-based sys-tems for specific uses could be hopefully available.From the results obtained along the developmenttime and after the execution of the test it can be de-duced that, nowadays, the RTE systems could beused with a remarkable success to identify informa-tion redundancy in multi-document summarizationtasks.Other applications for RTE systems should be ex-plored, such as automatic Answer Validation tasks.An example for this kind of task is proposed withinthe Cross Language Evaluation Forum (CLEF) forthe year 20062.7 AcknowledgmentsWe are grateful to Fernando López-Ostenero, fromUNED-NLP Group, for his named entities recog-nizer.This work has been partially supported by theSpanish Ministry of Science and Technology withinthe project: TIC-2003-07158-C04-02 MultilingualAnswer Retrieval Systems and Evaluation, SyEM-BRA.ReferencesI. Dagan, O. Glickman and B. Magnini. 2005. The PAS-CAL Recognising Textual Entailment Challenge. Pro-ceedings of the First PASCAL Recognizing TextualEntailment Workshop. LNAI. Springer. In press.S. Fischer, R. Klinkenberg, I. Mierswa and O. Ritthoff.2005. Yale 3.0, Yet Another Learning Environment.User Guide, Operator Reference, Developer Tutorial.University of Dortmund, Department of ComputerScience. Dortmund, Germany.J. Herrera, A. Peñas and F. Verdejo. 2005. TextualEntailment Recognition Based on Dependency Analy-sis and WordNet. Proceedings of the First PASCALRecognizing Textual Entailment Workshop. LNAI.Springer. In press.T. Joachims. 1998. Text Categorization with SupportVector Machines: Learning with Many Relevant Fea-tures. LS-8 Report 23. University of Dortmund, De-partment of Computer Science. Dortmund, Germany.D. Lin. 1998. Dependency-based Evaluation of MINI-PAR. Workshop on the Evaluation of Parsing Systems,Granada, Spain, May, 1998.V. Peinado, F. López-Ostenero, J. Gonzalo andF. Verdejo. 2005. UNED at ImageCLEF 2005: Au-tomatically Structured Queries with Named Entitiesover Metadata. Cross Language Evaluation Forum,Working Notes for the CLEF 2005 Workshop. LNCS.Springer. In press.M. Voorhees and D. Harman. 1999. Overview of theseventh text retrieval conference. Proceedings of theSeventh Text Retrieval Conference (TREC-7). NISTSpecial Publication.2http://nlp.uned.es/QA/AVE/

Spanish

UNED at PASCAL RTE-2 challenge

https://docta.ucm.es/bitstreams/d7dbc4b8-03b9-47f6-8fb9-5b19990644f7/download

UNED at PASCAL RTE-2 challenge

Abstract

Similar works

Full text

Available Versions

Docta Complutense