The Nobel process for science are often somewhat controversial for who they omit. A posthumous Nobel honor could help recognize some neglected heroes

D. Santos

H. Helbig

J. Leveling

S. Hartrumpf

English

PubMed

This paper describes the GIRSA-WP system and the experiments\ud
performed for GikiCLEF 2009, the geographic information retrieval task in the question answering track at CLEF 2009. Three runs were submitted. The first one contained only results from the InSicht QA system; it showed high precision, but low recall. The combination with results from the GIR system GIRSA increased recall considerably, but\ud
reduced precision. The second run used a standard IR query, while the third run combined such queries with a Boolean query with selected keywords. The evaluation showed that the third run achieved significantly higher mean average precision (MAP) than the second run. In both cases,\ud
integrating GIR methods and QA methods was successful in combining their strengths (high precision of deep QA, high recall of GIR), resulting in the third-best performance of automatic runs in GikiCLEF. The overall performance still leaves room for improvements. For example, the multilingual approach is too simple. All processing is done in only one\ud
Wikipedia (the German one); results for the nine other languages are collected by following the translation links in Wikipedia

Hartrumpf, Sven

Leveling, Johannes

Name not available

Recursive Question Decomposition forAnswering Complex Geographic QuestionsSven Hartrumpf1 and Johannes Leveling21 Intelligent Information and Communication Systems (IICS),University of Hagen (FernUniversita¨t in Hagen),Hagen, Germanysven.hartrumpf@fernuni-hagen.de2 Centre for Next Generation Localisation (CNGL)School of ComputingDublin City UniversityDublin 9, Irelandjleveling@computing.dcu.ieAbstract. This paper describes the GIRSA-WP system and the exper-iments performed for GikiCLEF 2009, the geographic information re-trieval task in the question answering track at CLEF 2009. Three runswere submitted. The first one contained only results from the InSicht QAsystem; it showed high precision, but low recall. The combination withresults from the GIR system GIRSA increased recall considerably, butreduced precision. The second run used a standard IR query, while thethird run combined such queries with a Boolean query with selected key-words. The evaluation showed that the third run achieved significantlyhigher mean average precision (MAP) than the second run. In both cases,integrating GIR methods and QA methods was successful in combiningtheir strengths (high precision of deep QA, high recall of GIR), result-ing in the third-best performance of automatic runs in GikiCLEF. Theoverall performance still leaves room for improvements. For example, themultilingual approach is too simple. All processing is done in only oneWikipedia (the German one); results for the nine other languages arecollected by following the translation links in Wikipedia.1 IntroductionGIRSA-WP (GIRSA for Wikipedia) is a fully-automatic, hybrid system com-bining methods from question answering (QA) and geographic information re-trieval (GIR). It merges results from InSicht, an open-domain QA system [1],and GIRSA, a system for textual GIR [2]. GIRSA-WP has already participatedin the preceding pilot task, GikiP 2008 [3, 4], and was improved based on thisand other evaluations.2 System Description2.1 GIRSA-WP SubsystemsThe GIRSA-WP system used for GikiCLEF 2009 integrates two basic systems:a deep (text-semantic) QA system (InSicht) and a GIR system (GIRSA, GIRwith semantic annotation). Each question is processed by both basic systems;GIRSA-WP filters their results semantically to improve precision and combinesboth result streams yielding a final result of Wikipedia article names, additionalsupporting article names (if needed), and supporting text snippets.The semantic filter checks whether the expected answer type (EAT) of thequestion and the title of a Wikipedia article are semantically compatible. Thistechnique is widely known from QA for typical answer types such as PERSON,ORGANIZATION, or LOCATION. In our system, a concept (a disambiguatedword) corresponding to the EAT is extracted from the question. The title ofeach candidate article is parsed by a syntactico-semantic parser for German [5].The resulting semantic representations (comprising the sort and the semanticfeatures, see [6] for details on the semantic representation formalism MultiNet)of the representations from the question and from the article title are unified. Ifthis unification succeeds, the candidate article is kept; otherwise it is discarded.For example, from topic GC-2009-06 (Which Dutch violinists held the post ofconcertmaster at the Royal Concertgebouw Orchestra in the twentieth century? ),the concept extracted as EAT is violinist.1.1, whose semantic representationbelongs to the class human (human-object in MultiNet). There are 87 such se-mantic classes, which can also be combined to form disjunctive expressions forunderspecification or for so-called semantic molecules (or semantic families).The retrieval in the GIR system works on the first few (two or three) sen-tences of the Wikipedia articles. Geographic names and location indicators (e.g.name variants and adjectives corresponding to toponyms) in the articles were au-tomatically annotated and normalized (see [2] for a discussion of this approach).As a result of our participation in GikiCLEF last year, we found that the fullWikipedia articles may be too long and indexing on a per-sentence basis doesnot provide enough context for matching. Therefore, we focused on the mostimportant parts of the Wikipedia articles (to increase precision for GIRSA), andchanged to full-document indexing.For the GikiCLEF 2009 experiments, the questions were analyzed by In-Sicht’s parser and sent to GIRSA and InSicht. In GIRSA, the top 1000 resultswere retrieved, with scores normalized to the interval [0, 1]. On average, GIRSAreturned 153 and 395 documents per question for run 2 and run 3, respectively(see Sect. 3). For results returned by both GIRSA and InSicht, the maximumscore was chosen (combMAX, [7]). Results whose score was below a given thresh-old were discarded and the semantic filter was applied to the remaining results.2.2 External KnowledgeTo obtain multilingual results, the German article names were ‘translated’ to thenine other languages using the Wikipedia linking between languages. Besides theinter-wiki links, GIRSA-WP uses one further information type from Wikipedia:the categories assigned to articles. Note that other Wikipedia information typeslike intra-wiki (i.e. inter-article) links and Internet links are still ignored.For the first time, two resources that contain structured information and arederived directly (categories) or indirectly (DBpedia) from Wikipedia were inte-grated into GIRSA-WP. The direct source of categories assigned to articles wasexploited by extracting categories from the Wikipedia XML file. The resultingrelations of the form in category(〈article title〉, 〈category〉) were reformulated inthe following form: 〈article title〉 ist ein/ist eine/ . . . 〈category〉/‘〈article title〉 isa . . . 〈category〉’. Some automatic corrections for frequent cases where the textwould be syntactically and/or semantically incorrect were implemented. The re-maining errors were largely unproblematic because the processing by InSicht’sparser detects them and avoids incorrect semantic networks. In this way, 1.1million semantic networks were generated for 1.5 million sentences derived fromaround 2 million in category relations.The DBpedia data is integrated in a similar way into GIRSA-WP by rephras-ing it in natural language. Specifically, version 3.2 of the file infobox de.nt, theinfobox information from the German Wikipedia encoded in N-Triples, a serial-ization of RDF was processed (see http://wiki.dbpedia.org/ for details). Asthere are many different relations in DBpedia, only some frequent and relevantrelations are covered currently. Each selected relation (currently 19) is associ-ated with an abstract relation (currently 16) and a natural language pattern.For example, the triple<http://dbpedia.org/resource/Andrea_Palladio><http://dbpedia.org/property/geburtsdatum>"1508-11-08"^^<http://www.w3.org/2001/XMLSchema#date>is translated to Andrea Palladio wurde geboren am 08.11.1508./‘Andrea Pal-ladio was born on 08.11.1508.’ This generation process led to around 460,000sentences derived from around 4,400,000 triples in the DBpedia file.The detour of translating structured information resources to natural lan-guage is performed with the goal to treat all resources in the same way, i.e.parsing them to obtain their representation as semantic networks. Hence, theresults can be used in the same way, e.g. for reasoning and to provide answersupport. In addition, the parser is able to resolve ambiguities; for example, namesreferring to different kinds of entities that had to be disambiguated explicitly onthe structured level otherwise.The QA system (InSicht) compares the semantic representation of the ques-tion and the semantic representations of document sentences. To go beyond exactmatching, InSicht applies many techniques, e.g. coreference resolution, query ex-pansion by inference rules and lexico-semantic relations, and splitting the querysemantic network at certain semantic relations. In the context of GikiCLEF, In-Sicht results (which are generated answers in natural language) must be mappedto Wikipedia article names; if this is not straightforward, the article name of themost important support is taken.2.3 Recursive Question DecompositionInSicht employed a new special technique called question decomposition (or querydecomposition, see [8] for details) for GeoCLEF 2007, GeoCLEF 2008, and GikiP2008. An error analysis showed that sometimes it is not enough to decomposea question once. For example, question GC-2009-07 (What capitals of Dutchprovinces received their town privileges before the fourteenth century? ) is de-composed into the subquestion Name capitals of Dutch provinces. and revisedquestion Did 〈SubA(nswer)1〉 receive its town privileges before the fourteenth cen-tury? Unfortunately, the subquestion is still too complex and unlikely to delivermany (if any) answers. This situation changes if one decomposes the subquestionfurther into a subquestion (second level) Name Dutch provinces. and a revisedquestion (second level) Name capitals of 〈SubA(nswer)2〉. InSicht’s processing ofquestion GC-2009-07 is illustrated in more detail in Fig. 1. For brevity and bet-ter readability, additional question reformulation phases and intermediate stageshave been omitted and the supporting texts are shortened and not translated.All subquestions and revised questions are shown in natural language, while thesystem operates mostly on the semantic (network) level.Question decomposition, especially in its recursive form, is a very powerfultechnique that can provide answers and justifications for complex questions.However, the success rates at each decomposition combine in a multiplicativeway. For example, if the QA system has an average success rate of 0.5, a doubledecomposition as described above (leading to questions on three levels) will havean average success rate of 0.125 (= 0.5 · 0.5 · 0.5).3 ExperimentsWe produced three runs with the following experiment settings:– Run 1: only results from InSicht.– Run 2: results from InSicht and GIRSA, using a standard query formulationand a standard IR model (tf-idf) in GIRSA.– Run 3: results from InSicht and GIRSA, using a Boolean conjunction of thestandard query formulation employed for GIRSA and (at most two) keywordsextracted from the topic.4 Evaluation and DiscussionInSicht achieved a higher precision than GIRSA: 0.7895 compared to 0.1076 and0.1442 for run 2 and run 3, respectively (see Table 2). The definition of theGikiCLEF score and other task details can be found in [9]. But InSicht’s lowrecall (only 30 correct answers compared to 107 and 142 correct answers for run2 and run 3, respectively) is still problematic as has already been seen in similarevaluations, e.g. GikiP 2008. As intended, InSicht aims for precision, GIRSA forrecall, and GIRSA-WP tries to combine both in an advantageous way.Table 1. Illustration of successful recursive question decomposition for topic GC-2009-07. The superscript designates the level of recursion, the subscript distinguishesalternatives on the same level of recursion.Q0 Welchen Hauptsta¨dten niederla¨ndischer Provinzenwurde vor dem vierzehnten Jahrhundert das Stadtrechtgewa¨hrt?‘What capitals of Dutch provinces received their townprivileges before the fourteenth century?’SubQ1 ← Q0 Nenne Hauptsta¨dte niederla¨ndischer Provinzen.‘Name capitals of Dutch provinces.’SubQ2 ← SubQ1 Nenne niederla¨ndische Provinzen.‘Name Dutch provinces.’SubA21 ← SubQ2 Zeeland (support from article 1530 : Besonders be-troffen ist die an der Scheldemu¨ndung liegendeniederla¨ndische Provinz Zeeland.)SubA22 ← SubQ2 Overijssel . . ....RevQ11 ← SubA21 + SubQ1 Nenne Hauptsta¨dte von Zeeland.‘Name capitals of Zeeland.’RevA11 ← RevQ11 Middelburg (support from article MiniatuurWalcheren: . . . in Middelburg, der Hauptstadt vonSeeland (Niederlande).; note that the orthographicvariants Zeeland/Seeland are identified correctly)SubA11 ← RevA11 Middelburg (note: answer to revised question can betaken without change)RevQ01 ← Q0 + SubA11 Wurde Middelburg vor dem vierzehnten Jahrhundertdas Stadtrecht gewa¨hrt?‘Did Middelburg receive its town privileges before thefourteenth century?’RevA01 ← RevQ01 Ja./‘Yes.’ (support from article Middelburg :1217 wurden Middelburg durch Graf Willem I. . . . dieStadtrechte verliehen.)A01 ← RevA01 + SubA11 Middelburg (support: three sentences, here from differ-ent articles, see supports listed in previous steps)In order to investigate the complementarity of GIRSA and InSicht, two ex-perimental runs were performed after the campaign. In run 4 and 5, only resultsfrom GIRSA are included; the settings correspond to the ones from run 2 andrun 3, respectively. The number of correct answers (compare run 2 and sum ofrun 1 and 4; compare run 3 and sum of run 1 and 5) shows that the overlapof GIRSA and InSicht is minimal: only 1 correct answer is shared. Hence, thecombination of both systems is very effective. The results indicate also that thecombination of the two systems profits from keeping most of InSicht’s correctresults and discarding some incorrect results from GIRSA.Table 2. Evaluation results for the three official GIRSA-WP runs and two experimentalruns.Run System Answers Correct answers Precision GikiCLEF score1 InSicht 38 30 0.7895 24.75832 InSicht+GIRSA 994 107 0.1076 14.51903 InSicht+GIRSA 985 142 0.1442 23.39194 GIRSA 964 78 0.0809 7.82595 GIRSA 961 113 0.1176 15.0473We made the following general observations:Complexity of Questions GikiCLEF topics are open-list questions and do notinclude factoid or definition questions. On average, GikiCLEF questions seem tobe harder than QA@CLEF questions from the years 2003 till 2008. Especiallythe presence of temporal and spatial (geographical) constraints in GikiCLEFquestions poses challenges for QA and GIR techniques, which cannot be metsuccessfully by shallow (i.e. syntactically-oriented) natural language processingor traditional IR techniques alone.Combination of standard and Boolean IR As the GikiCLEF topics resemble openlist questions, the aim of the GIR approach was to retrieve results with a highinitial precision. The use of the query formulation which combines keywordsextracted from the query with a standard IR query (run 3) increases preci-sion (+34%) and recall (+33%) compared to the standard IR query formulation(run 2).Question decomposition As our question decomposition experiments indicate,correct answers can often not be found in one step; instead, subproblems mustbe solved or subquestions must be answered in the right order. For some topics,a promising subquestion leads to many answers (for example, the subquestionNenne Sta¨dte in Deutschland./‘Name cities in Germany.’ for topic GC-2009-47), which cannot be efficiently handled for the revised questions so that correctanswers are missed.Abstract indexing Indexing shorter (abstracted) Wikipedia articles returned ahigher number of correct results (which was tested on some manually annotateddata before submission). Similarly, the annotation of geographic entities in thedocuments (i.e. conflating different name forms etc.) ensured a relatively highrecall.Multilingual Results The system’s multilingual approach is too simple becauseit relies only on the Wikipedia in one language (German) and adds results byfollowing title translation links to other languages. For eleven GikiCLEF topics(5, 10, 15, 16, 18, 24, 26, 27, 28, 36, and 39) no articles in German were assessedas relevant. Therefore for questions that have no or few articles in German,relevant articles in other languages cannot be found. Processing the Wikipediaarticles in parallel for another language in the same way also will allow to findsubanswers supported by articles in other languages, i.e. the supporting textsmay not only be distributed among different articles of only one languages, butalso among articles in different languages.5 Future WorkSome resources are not yet exploited to their full potential. For example, al-most half of the category assignments are ignored (see Sect. 2). Similarly, manyattribute-value pairs from infoboxes in DBpedia are not covered by GIRSA-WPcurrently. The cross-language aspect should be improved by processing at leastone more Wikipedia version, preferably the largest one: the English Wikipedia.AcknowledgmentsThis material is based upon works supported by the Science Foundation Irelandunder Grant No. 07/CE/I1142.References1. Hartrumpf, S.: Question answering using sentence parsing and semantic networkmatching. In Peters, C., Clough, P., Gonzalo, J., Jones, G.J.F., Kluck, M., Magnini,B., eds.: Multilingual Information Access for Text, Speech and Images: 5th Work-shop of the Cross-Language Evaluation Forum, CLEF 2004. Volume 3491 of LNCS.Springer, Berlin (2005) 512–5212. Leveling, J., Hartrumpf, S.: Inferring location names for geographic informationretrieval. In Peters, C., Jijkoun, V., Mandl, T., Mu¨ller, H., Oard, D.W., Pen˜as, A.,Petras, V., Santos, D., eds.: Advances in Multilingual and Multimodal InformationRetrieval: 8th Workshop of the Cross-Language Evaluation Forum, CLEF 2007.Volume 5152 of LNCS. Springer, Berlin (2008) 773–7803. Santos, D., Cardoso, N., Carvalho, P., Dornescu, I., Hartrumpf, S., Leveling, J.,Skalban, Y.: Getting geographical answers from Wikipedia: the GikiP pilot atCLEF. In: Results of the CLEF 2008 Cross-Language System Evaluation Campaign,Working Notes for the CLEF 2008 Workshop, Aarhus, Denmark (September 2008)4. Santos, D., Cardoso, N., Carvalho, P., Dornescu, I., Hartrumpf, S., Leveling, J.,Skalban, Y.: GikiP at GeoCLEF 2008: Joining GIR and QA forces for query-ing Wikipedia. In: Evaluating Systems for Multilingual and Multimodal Infor-mation Access: 9th Workshop of the Cross-Language Evaluation Forum, CLEF2008, Aarhus, Denmark, September 17–19, Revised Selected Papers. Volume 5706of LNCS. Springer, Berlin (2009) 894–9055. Hartrumpf, S.: Hybrid Disambiguation in Natural Language Analysis. Der AndereVerlag, Osnabru¨ck, Germany (2003)6. Helbig, H.: Knowledge Representation and the Semantics of Natural Language.Springer, Berlin (2006)7. Fox, E.A., Shaw, J.A.: Combination of multiple searches. In: The Second Text RE-trieval Conference (TREC-2). NIST Special Publication 500-215, National Institutefor Standards and Technology (1994) 243–2528. Hartrumpf, S.: Semantic decomposition for question answering. In Ghallab, M.,Spyropoulos, C.D., Fakotakis, N., Avouris, N., eds.: Proceedings of the 18th Eu-ropean Conference on Artificial Intelligence (ECAI), Patras, Greece (July 2008)313–3179. Santos, D., Cabral, L.M.: GikiCLEF: Expectations and lessons learned. This vol-ume.

Recursive question decomposition for answering complex geographic questions

Crossref

Recursive Question Decomposition for Answering Complex Geographic Questions

This paper describes the GIRSA-WP system and the experiments

performed for GikiCLEF 2009, the geographic information retrieval task in the question answering track at CLEF 2009. Three runs were submitted. The first one contained only results from the InSicht QA system; it showed high precision, but low recall. The combination with results from the GIR system GIRSA increased recall considerably, but

reduced precision. The second run used a standard IR query, while the third run combined such queries with a Boolean query with selected keywords. The evaluation showed that the third run achieved significantly higher mean average precision (MAP) than the second run. In both cases,

integrating GIR methods and QA methods was successful in combining their strengths (high precision of deep QA, high recall of GIR), resulting in the third-best performance of automatic runs in GikiCLEF. The overall performance still leaves room for improvements. For example, the multilingual approach is too simple. All processing is done in only one

Wikipedia (the German one); results for the nine other languages are collected by following the translation links in Wikipedia

DCU Online Research Access Service

Combination of multiple searches. In:

Getting geographical answers from Wikipedia: the GikiP pilot at CLEF. In: Results of the CLEF

Hybrid Disambiguation in Natural Language Analysis. Der Andere Verlag, Osnabr uck,

Inferring location names for geographic information retrieval.

Knowledge Representation and the Semantics of Natural Language.

L.M.: GikiCLEF: Expectations and lessons learned. This volume.

Question answering using sentence parsing and semantic network matching.

Semantic decomposition for question answering.

Eolene Boyd-MacMillan

MUCC (Crossref)

Transformation

This paper describes the GIRSA-WP system and the experiments
performed for GikiCLEF 2009, the geographic information retrieval task in the question answering track at CLEF 2009. Three runs were submitted. The first one contained only results from the InSicht QA system; it showed high precision, but low recall. The combination with results from the GIR system GIRSA increased recall considerably, but
reduced precision. The second run used a standard IR query, while the third run combined such queries with a Boolean query with selected keywords. The evaluation showed that the third run achieved significantly higher mean average precision (MAP) than the second run. In both cases,
integrating GIR methods and QA methods was successful in combining their strengths (high precision of deep QA, high recall of GIR), resulting in the third-best performance of automatic runs in GikiCLEF. The overall performance still leaves room for improvements. For example, the multilingual approach is too simple. All processing is done in only one
Wikipedia (the German one); results for the nine other languages are collected by following the translation links in Wikipedia

Irish Universities

https://genomebiology.biomedcentral.com/track/pdf/10.1186/gb-2006-7-10-117?site=genomebiology.biomedcentral.com

Transformation

Abstract

Similar works

Full text

Available Versions

Name not available

Crossref

DCU Online Research Access Service

MUCC (Crossref)

Irish Universities