Deleu, Johannes

Demeester, Piet

Demeester, Thomas

Develder, Chris

Mertens, Laurent

English

Ghent University Academic Bibliography

Context-Based Person Identificationfor News CollectionsLaurent Mertens, Thomas Demeester, Johannes Deleu,Chris Develder, Piet DemeesterINTEC - IBCNGhent University, Ghent, Belgiumfirstname.lastname@intec.ugent.beABSTRACTIn modern automated information extraction systems, NamedEntity Disambiguation (NED) techniques are becoming in-creasingly important. The ambiguity of person names leadsto a decrease in the output quality of search engines. Thispaper presents a two-stage rule-based NED model, basedon a local and global context of the mentioned persons. Anumber of experiments with different scoring functions arereported, as well as a specific evaluation method to estimatethe efficiency of the model on a real-life data collection inan unsupervised way.Categories and Subject DescriptorsH.3.3 [Information Storage and Retrieval]: InformationSearch and RetrievalGeneral TermsNamed Entity Disambiguation, Clustering1. INTRODUCTION AND BACKGROUNDIn the research areas of Natural Language Processing andInformation Extraction, Named Entities are entities that arereferred to by a proper name, such as persons and organi-sations. These entities are particularly relevant to searchengines, as they often correspond to query keywords. Thiskeyword property also leads to other important applicationareas, such as the development of the Semantic Web, andInformation Extraction techniques in general. A lot of re-search has been done on automatic Named Entity Recog-nition (NER), see e.g. [6] for a good overview of the earlywork on NER.The goal of NED is to resolve all Named Entities found in atext to a unique “real” instance, hereby correctly clusteringthe different forms as which an entity appears (e.g., only thefirst or last name of a person), also called “surface forms”,and taking into account the fact that one mention can re-fer to several entities (e.g., distinct people sharing the samename). Several NED techniques have been proposed in lit-erature, and many researchers have explored the potentialof Wikipedia as a means of evaluating such a system. Forexample, Bunesco and Pasca [1] used the Wikipedia con-tent and structure to train an SVM-based NED engine. Avector space model for NED was proposed by Cucerzan [2],DIR 2012 23–24 February 2012, Ghent, Belgiumstrongly based on Wikipedia as well. He also provides anice overview of earlier work on coreference resolution andNED. Further contributions along the same line, i.e., alsobased on Wikipedia, are [7] and [5]. A different approachis presented by Habib and Van Keulen [4], who explore thepossible interplay between extraction and disambiguation ofNamed Entities as a means to improve accuracy.This paper presents a number of experiments involving arule-based NED technique tailored towards a collection ofnews articles. We focus our attention on person names, andthe NED process, henceforth called the IDDiscriminator,boils down to mapping each mention of a person in the col-lection to the best matching entry in a gazetteer, an exter-nal reference repository, denoted as the RefIDBank. Notethat this external list determines the resolution of the ID-Discriminator, in the sense that distinct persons sharing thesame name can, at present, not correctly be distinguished.In order to determine the best match, the“context”of the ar-ticle in which a surface form occurs, is compared to a typicalcontext of a particular candidate. This context is consideredon two different levels. First, the Named Entities most oftenco-occuring with the considered surface form and candidatematch are taken into account. Secondly, the textual contextof the article is compared to the typical local textual contextof the candidate matches. The general workflow of the ID-Discriminator is depicted in Fig. 1, the different componentsof which are explained in Section 2. Section 3 presents someexperiments, in particular focusing on a new non-supervisedevaluation method. Finally, Section 4 summarises the mainresults.2. NED MODEL2.1 Data Preparation2.1.1 Data CollectionThe raw data consists of a dataset of news articles fromDutch newspapers, spanning a time range from March 1st2011 untill August 30th 2011, and totalling 353798 items.This data has been provided by Flemisch media consortiumMediargus (http://www.mediargus.be).2.1.2 The Named Entity RecogniserFirst, the articles are parsed by a Named Entity Recog-niser [3]. Four categories are distinguished by the NER en-gine: persons (PER), organisations (ORG), locations (LOC)and miscellaneous (MISC). The total number of distinctPER surface forms in the entire dataset amounts to 264823.NERWikipediaORGMapRefIDBankIDDiscriminatorDataset(ContextBank)NEDDataprocessingInputConfidentInnerIDsGazetteerFigure 1: Graphical workflow overview.2.1.3 The InnerIDsBefore resolving names over the entire dataset, the PERsare clustered per article. The longest item per such clusteris referred to as an InnerID, and the cluster will contain allsubstrings of this InnerID that appear in the article. Sup-pose, e.g., an article contains the PERs “Bart”, “De Wever”,“Bart De Wever” and “Di Rupo”, then the first three willdefine one cluster, while the last one will form a cluster onits own. The two InnerIDs of this article are then “Bart DeWever” and “Di Rupo”. Note that this leads to a reductionin the number of distinct original surface forms with only5%.2.1.4 The RefIDBankThe aim of the IDDiscriminator is to look in an externalrepository for the best match to a certain mention of a per-son. This repository is called the RefIDBank, and its entriesare the RefIDs.The candidate RefIDs are gathered from two sources, namelyWikipedia, and a list of “confident entities”, i.e., entities forwhich the NER engine label assignment confidence exceedsa certain threshold, taken from the NER output [3] for thenews dataset. For each candidate that appears as an In-nerID in at least one article, the application will gather alist of the x most frequent PERs and ORGs that co-occurwith this particular name in the dataset, by analysing thearticles in which it appears as an InnerID, and where x is auser-defined parameter. As far as the PERs are concerned,only “full names” are considered, i.e., names that are com-posed of more than 1 word, and PERs that are also RefIDsare given more weight. For the ORGs, the “ORGMap” ideaas described in 2.1.5 is applied. The resulting most com-monly co-occuring x entities (or less if there aren’t enough),are then listed in descending order of co-occurence frequencyin the “RefIDBank of length x”, one entry of which is shownin Fig. 2.2.1.5 The ORGMapOften, several surface forms of the same organisation ap-pear in the data. For example, Coca-Cola might appear as“Coca-Cola”, “Coca Cola”, “Coke”, etc. When one simplyElio Di Rupo:ORGs: N VA PS CD&V VLD [...]PERs: Bart De Wever Wouter Beke [...]Figure 2: One entry from the RefIDBank.determines the organisations that co-occur most frequentlywith a certain person, all of these different variations will beconsidered to be different entities as well, while in fact, theyall refer to the same organisation. In order to deal with thisissue, the ORGs are converted to a normalised writing, thenclustered, and finally mapped to the shortest common partthey share.The “normalising” step consists of converting every ORG toa standard form by applying a number of formatting rules.Subsequently, the normalised organisations are grouped intoclusters that start or end with a same common shortestORG, which is used to identify the cluster. For example,the cluster identified by“KIWANIS”contains items like“KI-WANIS ANTWERPEN” and “JONG KIWANIS”. Finally,each normalized ORG is mapped onto the identifier of thecluster in which it is contained. This mapping is called the-ORGMap, and in the RefIDBank the ORGs are representedby these identifiers.2.1.6 The ContextBankThe ContextBank is a document that lists, per RefID, thosewords that frequently appear within a certain “context win-dow” around mentions of that RefID. The way this docu-ment is obtained, goes as follows: for all mentions of theRefIDs in the dataset, gather all words within the contextwindow, and count how many times each word is observed.Once these data have been gathered, each RefID is scoredby normalising these counts by the total number of times itappears in the dataset. Finally, retain those words with ascore > 0.001, a well-chosen cutoff value.The ContextBank used for the runs described in this articlewere obtained with a context window of 3 words precedingand following the mentions of the RefIDs. Note howeverthat its use can be turned off, which we use to evaluate itssignificance.2.2 Person Identification AlgorithmNow that the data preparation steps for the IDDiscriminatorhave been described, the main algorithm, mapping InnerIDson RefIDs, is succinctly described.2.2.1 Algorithm OverviewThe pseudocode for the algorithm is depicted in Figure 3.First, the InnerIDs for all documents are computed, i.e., allPERs are clustered per article, thereby taking into accountpossible spelling mistakes.In a second step, the algorithm proceeds to finding the bestmatch in the RefIDBank for every InnerID in every docu-ment. For this, it is first verified if the InnerID is also aRefID, in which case it is assumed that it is its own bestmatch. If this is not the case, the set of all RefIDs thatcontain the InnerID as a subset will be gathered, and thecontext of the document will be compared to the contextIDDiscriminator(D,R):1. for doc ∈ DInnerIDs[doc] = ComputeInnerIDs(doc);2. for doc ∈ D:for ID ∈ InnerIDs[doc]:if ID ∈ R:BestMatch = ID;else:RefIDMatches = GatherMatches(ID);BestMatch = null;if !RefIDMatches.isEmpty():for RefID ∈ RefIDMatches:Scores[RefID] = ComputeScore(RefID);if max Scores[RefID] > 0:BestMatch = argmax Scores[RefID]return BestMatch;Figure 3: Pseudocode for IDDiscriminator. The ar-guments are the dataset D and the RefIDBank R.of these RefIDs, as stored in the RefIDBank, and option-ally also the ContextBank. This comparison will result in ascore, and the RefID with the highest score will be taken asthe best match for the considered InnerID.2.2.2 Scoring FunctionThe scoring proceeds as follows. For a given InnerID in agiven document d, and for every candidate RefID match,the (other) InnerIDs and ORGs contained in that documentwill be compared with the PERs and ORGs stored in theRefIDBank for that RefID. Starting from 0, the score willbe incremented for every match between both lists. Definea “full match” as one where an InnerID appears as such inthe RefIDBank, as opposed to a “partial match” where itappears “contained within” a PER listed in the RefIDBank(e.g., “Bart” is a partial match for“Bart De Wever”). Partialmatches only count once, e.g., suppose an InnerID appearsas a subset of 3 different PERs, only the first one (the onewith the lowest position) is accounted for. For ORGs, onlyfull matches are considered.Five different scoring functions were experimented with. Theseare depicted in Table 1. Note that positions are countedstarting from 0, up till N − 1, where N is the length ofthe RefIDBank, and that PERs are compared before ORGs.The used abbreviations correspond to: normalised (Norm),One-over-X (OOX), One-over-square-root (OOSR), LinearDown (LinDn) and Linear Up (LinUp). Note that the ideawas not to design an optimised scoring function, as theweights are still chosen arbitrarily, but rather to evaluatedifferent tendencies.If the system is used with the ContextBank, the score will beadditionally incremented with the score for every word thatappears both in the entire document and the ContextBankentry for the candidate RefID, and this regardless of thescoring function used.3. EXPERIMENTS & RESULTSFull PartialPER Match PER Match ORG MatchNorm +2 +1 ×2OOX +2 (X + 1)−1 +(X + 1)−1 ×2OOSR +2 (X + 1)−1/2 +(X + 1)−1/2 ×2LinDn +2 (1−X/N) + (1−X/N) ×2LinUp +2(X + 1)/N +(X + 1)/N ×2Table 1: Used scoring functions, with X the positionof the match in the RefIDBank vector, and N thelength of the RefIDBank.To test the accuracy of the IDDiscriminator, a number ofexperiments were carried out.1 For these tests, several newdatasets were created from the original dataset. The idea isto take names of which one is confident, i.e., the RefIDs, takeall the articles in which these names appear (as InnerID),and replace the surface forms of these InnerIDs at randomwith either only the first or only the last name of this RefID.2This way, one knows in advance what RefID these particu-lar mentions should be resolved to by the IDDiscriminator,and furthermore creates a worst case scenario, in which noRefIDs appear as such anymore, and hence all surface formsneed to be resolved using the context information.Five different sets were created, using the x RefIDs thatappear most frequently, with x ∈ {50, 100, 150, 200, 250},and keeping only the articles in which they appear. Thesedatasets will hitherto collectively be referred to as the “TopX datasets”. How many documents each of these datasetscomprise, can be seen in Table 2.#Top RefIDs # Docs % Total Docs50 25047 7.1100 34647 9.8150 40080 11.3200 43493 12.3250 46742 13.2Table 2: Size of the Top X datasets.3.1 Evaluation MethodThe output of the clustering tool is a document that specifiesfor each surface form to which InnerID, which possibly is alsoa RefID, it gets mapped. But for the Top X datasets, weknow in which documents these RefIDs originally appear,and have replaced each explicit RefID mention by only itsfirst or last name. As such, after the cluster tool has doneits work, it is possible to verify for each of these Top XRefIDs in how many of the documents in which they shouldappear, the tool correctly predicted the right mapping. Bydividing the number of correctly resolved documents by thenumber of documents in which the RefID should appear,one obtains a score from 0 to 1, indicating the resolvingefficiency of the tool for that particular RefID. The score for1Note that NED techniques are often evaluated using thedisambiguation pages of Wikipedia. However, this is notpossible here, given the insufficient overlap between the usednews archive and Wikipedia in terms of persons.2By “first name” we understand the first word, i.e., every-thing preceding the first whitespace, of a “full name”.a Top X dataset is then the average of these scores takenover all Top X RefIDs. In the subsequent paragraphs, thisscore is referred to as “accuracy”.3.2 Scoring FunctionsFirst of all, it was verified which of the scoring functionsdepicted in Table 1 performed best. Runs using these dif-ferent functions were executed on the Top X datasets, forRefIDBank lengths 10, 20, 30, 40 and 50, without usingthe ContextBank. The results, including the running times,are shown in Table 3. It can be seen that the OOSR func-tion performed best, although not by much, and hence thisis the function that is used in the IDDiscriminator for allother runs discussed in this article.Norm OOX OOSR LinDn LinUpTop 500.572 0.548 0.583 0.583 0.5241:49:35 1:52:12 1:49:20 2:05:05 1:59:33Top1000.590 0.566 0.597 0.595 0.5433:19:31 3:08:12 3:27:00 3:09:35 3:08:27Top1500.598 0.571 0.602 0.601 0.5533:53:08 3:08:45 4:18:02 3:55:03 4:24:59Top2000.612 0.584 0.614 0.613 0.5673:29:32 3:56:51 3:38:18 4:35:56 4:03:56Top2500.620 0.590 0.621 0.620 0.5794:06:25 4:08:08 3:45:08 3:47:37 4:09:50Table 3: Accuracy and running time (in h:mm:ss)for Top X datasets for different scoring functions,all runs with (N = 20), and no context.3.3 Top RefID DatasetsA further set of runs was performed on the Top RefID datasetsdescribed earlier. For every dataset, 10 runs were performed,namely two runs per RefIDBank lenghts 10, 20, 30, 40, 50and 100. One run did not use the ContextBank, whereas theother one did. The results of these runs are shown in Fig. 4.Taking into account the context clearly leads to better re-sults. The accuracy also increases when taking into accountthe context of co-occuring persons and organizations, i.e., foran increasing vector size of the RefIDBank. This increaseappears to plateau if one were to carry on experiments withstill higher vector sizes. It can also be observed that theadded benefit of the extra context becomes less importantas the length of the RefIDBank increases, because the longerthe RefIDBank, the less extra relevant information is con-tained in the remaining context. Note however, that thereis still a 6% increase in accuracy for a RefIDBank length of100, which remains very significant.3.4 ORGMap vs. No ORGMapAdditional runs without ContextBank were executed thatdid not employ the ORGMap principle, for all TopX datasets,and RefIDBank lengths 10, 20, 30, 40 and 50. These runs in-dicated that the ORGMap barely had any positive effect forthe Top 50 dataset, with an average increase in accuracy of0.4%, but that for bigger datasets the difference does becomesignificant, with an average accuracy increase of 2.0% overthe remaining 4 datasets. The effect however strongly de-creased from the smallest to the largest RefIDBank lengths,with respective average accuracy increases of 3.0± 0.3% vs.1.3± 0.1%.0.60.7  AccuracyRefIDBank lengthwith ContextBankwithoutContextBankTop 50Top 100Top 150Top 200Top 2500 50 1000.50.8Figure 4: Influence of context on accuracy for 5 dif-ferent dataset sizes4. CONCLUSIONSWe presented a rule-based Named Entity Disambiguationmodel for persons, where the different occuring surface formsare mapped to the entries of a reference gazetteer, based ona local textual context and a global context of co-occuringpersons and organisations. The influence of these contextualfactors was demonstrated in a number of experiments, basedon a basic but efficient evaluation method. Future workconsists of investigating the effect of the size of the contextwindow, as well as optimising the scoring function.5. REFERENCES[1] R. Bunescu and M. Pas. Using EncyclopedicKnowledge for Named Entity Disambiguation.Computational Linguistics, (April):9–16, 2006.[2] S. Cucerzan. Large-Scale Named EntityDisambiguation Based on Wikipedia Data.Computational Linguistics, (June):708–716, 2007.[3] J. Deleu, B. Vermeulen, A. De Moor, T. Demeester,and B. Van Den Bossche. Building a dedicated NamedEntity Recognizer for Dutch Multimedia Archives. InProceedings of the 12th Dutch-Belgian InformationRetrieval Workshop, 2012.[4] M. Habib and M. van Keulen. Named Entity Extractionand Disambiguation: The Reinforcement Effect. InProceedings of the 5th International Workshop onManagement of Uncertain Data, volume WP11-02,pages 9–16. Centre for Telematics and InformationTechnology, University of Twente, Aug. 2011.[5] X. Han and J. Zhao. Named entity disambiguation byleveraging wikipedia semantic knowledge. In Proceedingof the 18th ACM conference on Information andknowledge management - CIKM ’09, page 215, NewYork, New York, USA, Nov. 2009. ACM Press.[6] D. Nadeau and S. Sekine. A survey of named entityrecognition and classification. LingvisticaeInvestigationes, 30(1):3–26, 2007.[7] H. Nguyen and T. Cao. Named entity disambiguation:A hybrid statistical and rule-based incrementalapproach. The Semantic Web, pages 420–433, 2008.

Context-based person identification for news collection

https://biblio.ugent.be/publication/2911556/file/2911573.pdf

Context-based person identification for news collection

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography