Desmet, Bart

Hoste, Veronique

English

Ghent University Academic Bibliography

Towards a Balanced Named Entity Corpus for DutchBart Desmet1 ,2 and Ve´ronique Hoste1 ,21LT3, Language and Translation Technology Team, University College GhentGroot-Brittannie¨laan 45, 9000 Gent, Belgiumbart.desmet, veronique.hoste@hogent.be2Department of Applied Mathematics and Computer Science, Ghent UniversityKrijgslaan 281 (S9), 9000 Gent, BelgiumAbstract1. IntroductionNamed Entity Recognition (NER) is the task of auto-matically identifying names within text and classifyingthem into categories, such as persons, locations andorganizations. NER started as an information extractionsubtask, but has since evolved into a distinct task essentialfor information retrieval, question answering, and as apreprocessing step for coreference resolution and variousother problems. An extensive literature on the subject ex-ists (see for example (Tjong Kim Sang, 2002), (Chinchor,1998)), with NER approaches roughly falling into threecategories: hand-crafted, machine learning and hybridsystems. Hand-crafted approaches use gazetteer lists andrequire manual rule creation, a time-consuming processwhich hinders easy porting to new domains or languages.Supervised machine learning solutions, on the other hand,rely on an annotated training corpus to infer patternsassociated with named entities, based on orthographic,syntactic, lexical and contextual features. Hybrid systemscombine both approaches. Such systems are in widespreaduse and have proven their effectiveness, with (Zhou andSu, 2002) reporting near-human performance on Englishdata.The bottleneck for the development of machine learningapplications is its dependence on, preferably large, anno-tated training corpora. Named entity resources for Englishinclude the manually annotated data sets from the MUC-7Named Entity Task ((Chinchor, 1998), 162,692 tokens),the CoNLL-2003 shared task ((Tjong Kim Sang andDe Meulder, 2003), 301,418 tokens) and the BBN PronounCoreference and Entity Type Corpus ((Weischedel andBrunstein, 2005), 1,173,766 tokens). For Dutch, however,the data from the CoNLL-2002 shared task ((TjongKim Sang, 2002), 309,686 tokens from four editions of theBelgian newspaper ”De Morgen” of 2000) constitute theonly corpus annotated with named entity information thatis readily available at present.The pressing need for a substantial corpus of Dutch text,not only for NER, is addressed in the STEVIN1-fundedSoNaR project2. It aims to produce a 500-million-wordreference corpus of written Dutch containing a wide1http://taalunieversum.org/taal/technologie/stevin/2http://lands.let.ru.nl/projects/SoNaR/spectrum of genres and text types (Oostdijk et al., 2008),including a 1-million-word subset with a number of man-ually corrected annotation layers, including four semanticones: named entities, coreference relations, semanticroles and spatiotemporal expressions (see Schuurman etal., (2009)). The subset contains the various text types,reflecting the global corpus design. This diversity, whichwas particularly lacking in the Dutch CoNLL-2002 dataset, should allow for a more robust classifier and bettercross-corpus performance.For the named entity annotation of the corpus, we devel-oped new annotation guidelines, based on the guidelinesfrom MUC-7 (Chinchor and Robinson, 1997) and ACE(LDC, 2008). A number of adaptations were made, mostnotably the addition of separate classes for products andevents, and the annotation of metonymy.In the remainder of this abstract, we will discuss andmotivate the annotation guidelines in Section 2, we presentan evaluation of the guidelines based on inter-annotatoragreement scores in Section 3 and give an overview ofthe use of the guidelines within the context of the SoNaRproject in Section 4. Section 5 concludes this abstract.2. Annotation guidelinesThe SoNaR named entity annotation guidelines3 are basedon the MUC-7 and ACE annotation schemes for Englishnamed entities. Annotation of numerical and temporal ex-pressions was considered beyond the scope of the NE task.The aim of the new guidelines was to achieve consistentand fine-grained annotation of Dutch text. To that end, theguidelines describe the delimitation of named entities (see2.1.), the classification into main types (2.2.) and subtypes(2.3.), and the markup of metonymic usage (2.4.). For anoverview of the possible annotations, see Figure 1.2.1. SpanNamed entities are often defined as “unique identifiers ofreferents in reality”. In practice, it is often unclear whethera given phrase should be considered a unique identifier ornot. This is especially true for named entities of the types3http://lt3.hogent.be/sonar/share/AnnotatierichtlijnenNE20091019.pdf,in DutchFigure 1: Annotation scheme for named entities, with categories for main type, subtype, usage and metonymic roles.product, event and miscellaneous (see 2.2.). Consider thefollowing example:(1) Koning Albert zal de twee Koninklijke Besluiten vanminister van Werk Peter Vanvelthoven (sp.a) niet on-dertekenen.English: King Albert will not sign the two Royal De-crees by the minister of Employment, Peter Vanvelthoven(sp.a).In sentence 1, it is debatable whether “King”, “Royal De-crees” and “Employment” should be considered (part of) aunique identifier. For reasons of consistency, a pragmaticapproach was taken to the delimitation of named entities.All words starting with a capital letter that are not the firstword of a sentence are taken to be named entities. Allsentence-initial or uncapitalized words that can unequivo-cally be considered unique identifiers are annotated as well.Sentence 1 will be annotated as follows:(2) Koning [Albert] zal de twee [Koninklijke Besluiten] vanminister van [Werk] [Peter Vanvelthoven] ([sp.a]) nietondertekenen.English: King [Albert] will not sign the two [RoyalDecrees] by the minister of [Employment], [Peter Van-velthoven] ([sp.a]). (sp.a is a Belgian political party)Named entities can be part of a word that as a whole is not anamed entity, e.g. “London-based”. In English, such struc-tures are rare and will often be annotated fully as MISC,or not at all (Nothman et al., 2009). Given the frequencyof concatenated compounds in Dutch, we chose to annotatenamed entities word-internally:(3) [Apple]topman [Steve Jobs] kondigde het[iPhone]platform op [Macworld 2007] aan.English: [Apple] [CEO] [Steve Jobs] announced the[iPhone] platform at [Macworld 2007].2.2. Main typesTokens marked as named entities can be classified as one ofsix main types, namely person (PER), organization (ORG),location (LOC), product (PRO), event (EVE) and miscella-neous (MISC). PER, ORG and LOC are the usual suspectsin named entity annotation, with MISC sometimes actingas a backup class. We added the PRO and EVE categoriesto obtain good coverage of possible named entities and toallow for consistent metonymy roles (see 2.4.). The MISCcategory was reserved for instances produced by the broaddefinition of span (2.1.) that fitted more than one or none ofthe five other main types.2.3. SubtypesFor the ORG, LOC, PRO and EVE classes, mutually ex-clusive subtypes are to be annotated. The motivation forsubtypes was twofold:1. It allows for fine-grained annotation, as required forquestion answering tasks, without compromising a ro-bust, coarse-grained main type structure.2. It provides useful information for the classification ofusage (2.4.) and for the other semantic annotation lay-ers in the SoNaR project.The markables in Sentence 3 would be classified as follows:[Apple]ORG.comtopman [Steve Jobs]PER kondigdehet [iPhone]PRO.miscplatform op [Macworld2007]EV E.human aan.English: [Apple]ORG.com [CEO]MISC [SteveJobs]PER announced the [iPhone]PRO.misc platformat [Macworld 2007]EV E.human.2.4. UsageAn important issue we wanted to address in our annotationscheme was the metonymic use of named entities. ConsiderSentence 4:(4) Het [Witte Huis] koos voor moderne werken, waarondereen [Rothko].English: The [White House] opted for modern works ofart, including a [Rothko].Cases like “White House” being classified as LOC ratherthan ORG are a common mistake (Nothman et al., 2009).By marking whether a NE is used literally or metonymi-cally, we can consistently label named entities for their lit-eral main type, and use metonymic roles to point to theirintended main type (PER, ORG, LOC, PRO or EVE). Thisapproach was inspired by Markert and Nissim (2002) andMarkert and Nissim (2007). Because it is often impracti-cable to determine whether a NE is used metonymically asPER or as ORG, we combined them in the intended type“human” (see for example 5, where “White House” mightrefer to a PER, namely the U.S. President, or to an ORG-like group of people such as the White House staff). Whena name is used as a mere signifier, the intended type is“name”.(5) Het [Witte Huis]LOC.point.meto.humankoos voor moderne werken, waaronder een[Rothko]PER.meto.PRO.misc.English: The [White House]LOC.point.meto.humanopted for modern works of art, including a[Rothko]PER.meto.PRO.misc.Marking metonymy does not only do away with confusablemain types, it should also benefit the automatic annotationof other semantic layers. For example, a coreferential res-olution algorithm could link an inanimate noun phrase like“the painting” to “Rothko” in Sentence 5 if it has access toNE classifier output that does not only mark “Rothko” lit-erally as an (animate) person, but also metonymically as aproduct.3. Guideline evaluationIn order to evaluate the guidelines, two linguists annotateda set of eight randomly selected texts from the corpus,containing 14,244 tokens in total. Two evaluation metricswere used: Kappa (Carletta, 1996) and F-score (β = 1)(Van Rijsbergen, 1979). F-scores were calculated bytaking one annotator as the gold standard and scoring theannotations of the other for precision and recall. Thisyields the same results as averaging the precision scores ofboth annotators, when using the other as a gold standard.Scores were calculated on 5 levels: span, main type,subtype, usage and metonymic role. For each level, scoreswere calculated on the entire set, and on a subset containingonly those tokens (i) on which both annotators agreedon the preceding level, and (ii) which bore annotation onthe current level (MISC and PER, for example, are notincluded in the subset for subtype). The results can befound in Table 1, in which absolute counts for span, maintype and usage are also included.The results show high agreement scores for all levels, mostnotably span. However, both global metrics fail to indi-cate low agreement for minority classes, as is the case formetonymic usage. F-scores per class have been calculatedto detect problematic classifications and to refine the guide-lines accordingly.4. Annotation implementationThe SoNaR corpus will comprise a wide variety of texts,including newswire, manuals, autocues, online material,fiction and reports, for a total of 500 million words. Arepresentatively diverse 1-million-word subset is beingannotated manually, and serve as the gold standard for theautomatic annotation of the entire corpus. This diversityis essential to training automatic classifiers that can beapplied on the corpus as a whole, and should also make itan interesting corpus for research on domain adaptation(Nothman et al., 2009).Manual annotation is done using the MMAX2 annotationtool4. Each annotation layer is stored as one or severalstandoff XML files. For the named entity task, six XMLfiles are created - one per main type. Annotation speed av-erages around 4,000 words per hour. Taking into accountthe verification of the annotations by a second annotator,the actual annotation speed will be closer to 2,500 wordsper hour.5. Conclusion and future workThis abstract presented the named entity annotationguidelines for the SoNaR project. Their aim is to provideconsistent and fine-grained annotations that capture usefulinformation for subsequent classification tasks. To thisend, a pragmatic approach was taken for the delimitation ofnamed entities, resulting in high inter-annotator agreementscores for span.In the full version of this paper, we intend to provide statis-tics on the distribution of the various types, usage andmetonymic roles, and discuss the generalization perfor-mance of classifiers trained on the fully annotated corpus.6. ReferencesJ. Carletta. 1996. Assessing agreement on classificationtasks: The kappa statistic. Computational Linguistics,22(2):249–254.N. Chinchor and P. Robinson. 1997. MUC-7 named entitytask definition. In Proceedings of the 7th Conference onMessage Understanding.N. Chinchor. 1998. Overview of MUC-7. In Proceedingsof the 7th Message Understanding Conference.4http://www.eml-research.de/english/research/nlp/download/mmax.phpLevel Total set SubsetKappa Fβ=1 Kappa Fβ=1 Tokens DistributionSpan 0.97 99.62 0.97 99.62 14244 13293 non-NE, 897 NE, 54 NAMain type 0.94 99.23 0.92 93.76 897 150 PER, 225 ORG, 241 LOC, 115 PRO, 62 EVE, 48 MISC, 56 NASubtype 0.92 99.12 0.94 97.67 643 32 NAUsage 0.91 98.93 0.93 94.58 793 733 literal, 17 metonymic, 43 NARole 0.91 98.90 1.00 100.00 17 0 NATable 1: Inter-annotator agreement scores per level, token count and distribution (NA = no agreement).LDC, 2008. ACE (Automatic Content Extraction)English Annotation Guidelines for Entities Ver-sion 6.6. Linguistic Data Consortium, Philadelphia.http://projects.ldc.upenn.edu/ace/.K. Markert and M. Nissim. 2002. Towards a corpus an-notated for metonymies: the case of location names.In Proceedings of the Third International Language Re-sources and Evaluation (LREC’02), pages 1385–1392,Las Palmas, Spain.K. Markert and M. Nissim. 2007. SemEval-2007 task 08:Metonymy resolution at SemEval-2007. In Proceedingsof the 4th International Workshop on Semantic Evalua-tions (SemEval-2007), pages 36–41, Prague, Czech Re-public.J. Nothman, T. Murphy, and J.R. Curran. 2009. AnalysingWikipedia and gold-standard corpora for NER training.In Proceedings of the 12th Conference of the EuropeanChapter of the ACL, pages 612–620, Athens, Greece.N. Oostdijk, M. Reynaert, P. Monachesi, G. Van Noord,R. Ordelman, I. Schuurman, and V. Vandeghinste. 2008.From D-Coi to SoNaR: A reference corpus for Dutch.In Proceedings of the Sixth International Language Re-sources and Evaluation (LREC’08), Marrakech, Mo-rocco.I. Schuurman, V. Hoste, and P. Monachesi. 2009. Cultivat-ing trees: Adding several semantic layers to the Lassytreebank in SoNaR. In Proceedings of the 7th Interna-tional Workshop on Treebanks and Linguistic Theories,Groningen, The Netherlands.E.F. Tjong Kim Sang and F. De Meulder. 2003. Intro-duction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedingsof the 7th Conference on Natural Language Learning,pages 142–147, Edmonton, Canada.E.F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: Language-independent named entityrecognition. In Proceedings of the 6th Conference onNatural Language Learning, pages 155–158, Taipei, Tai-wan.C.J. Van Rijsbergen. 1979. Information Retrieval. Butter-worth, London.R. Weischedel and A. Brunstein, 2005. BBN PronounCoreference and Entity Type Corpus. Linguistic DataConsortium, Philadelphia, USA.G.D. Zhou and J. Su. 2002. Named entity recognition us-ing an HMM-based chunk tagger. In Proceedings of the40th Annual Meeting on Association for ComputationalLinguistics, pages 473–480, Philadelphia.

Towards a balanced named entity corpus for Dutch

https://biblio.ugent.be/publication/1055839/file/6719410

Towards a balanced named entity corpus for Dutch

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography