De Moor, An

Deleu, Johannes

Demeester, Piet

Demeester, Thomas

Vermeulen, Brecht

English

Ghent University Academic Bibliography

Named Entity Recognition on Flemish audio-visual andnewspaper archivesJohannes Deleu, An De Moor, Thomas DemeesterBrecht Vermeulen, Piet DemeesterINTEC - IBCN - IBBTGhent University, Ghent, Belgiumfirstname.lastname@intec.ugent.beABSTRACTThis paper describes a number of specific issues that weneeded to deal with, in order to compose an accurate NamedEntity Recognition tool on multimedia archives in Dutch.The considered data consists of archivation metadata fromvideo collections, and large newspaper collections. For thevideo collections, the main challenge is to cope with a lackof capitalization in the metadata. To this end, specific cap-italization features are calculated from Wikipedia. For thenewspaper collections, the main concern is to create a sys-tem that maintains its performance over the course of manyyears. For that goal, special clustering features allow deal-ing with words that have not been encountered in trainingdata. Results for the different components of the tool arereported on the target data, as well as on publicly availabletest data.Categories and Subject DescriptorsH.3.3 [Information Storage and Retrieval]: InformationSearch and Retrieval1. INTRODUCTIONWe present a Named Entity Recognition (NER) system thatwas initially developed to run on the data of the “Vlaan-deren in Beeld”(VLIB) project [4], involving several Flemishbroadcaster archives, and was further developed for an ex-tensive newspaper collection, in the framework of the “TEx-SIS” [3] project. Our main goal was to obtain a system withthe highest possible performance, given the (considerable)impediments of the data, the Dutch target language, andwith a wide variety of possible techniques to tweak the NERengine at our disposal. Therefore, we describe the maindesign and a number of particular properties of our NERengine. We present the results according to the evolution ofthe system. For reference, we also show its performance ona publicly available test collection.The Flemish research project VLIB was funded by the IWT(agency for Innovation by Science and Technology), and fo-cused on the creation of a prototype multimedia archive forthe Flemish broadcasters. The prototype archive containsroughly about 10.000 hours of video material, including au-dio data, and a large amount of textual metadata. Abouthalf of this material relates to news and contains a fairamount of textual metadata. The full text search functional-DIR 2012 23–24 February 2012, Ghent,Belgiumity for the archive was provided by indexing these metadata(i.e., textual descriptions and keywords).As the content of the archive was provided by different part-ners, an important step was to bring together the meta-data into a unified model. Due to the individual proce-dures of adding metadata, there was an important need fordata normalization, such as merging keywords from differentthesauruses together (e.g., Kongo / Zaire), correcting typo-graphical errors, and so on. Moreover, it appeared essentialto fill in missing data as a considerable amount of documentsdid not contain any keywords at all. We therefore startedlooking for ways to automaticaly extract information fromthe full text descriptions, and use these to fill in and correctthe metadata fields.Named Entity Recognition (NER) appeared to be an at-tractive means for that goal. It would enable the automaticextraction of names such as people, locations and organi-zations from the full text. Unfortunately, existing NamedEntity Recognizers did not perform well on our data. Mostsystems are trained on a collection of news articles. Thequality of metadata written for archivation purposes is typ-ically less critical. The text fragments in the multimediaarchive are often written in a very condensed form, usingdomain-specific terminology, and with a lot of editorial is-sues (such as the lack of capitalization). The new NER sys-tem was initially built to circumvent these problems with anumber of special measures.In a further stage, during the TExSIS project, the NERengine was used on a large data set from Mediargus [2],initially containing the Flemish newspapers’ content of 2011.Although the quality of this collection was clearly superior,the main challenge is to build a system that works reliablyover a longer period of time, as the number of new words isexpected increase quickly over time.The paper is outlined as follows. The next section describesthe architecture of the NER engine and highlights where itcan be most clearly distinguished from other systems. Sec-tion 3 describes a number of experiments and overal per-formance results. Section 4 describes possible future workdirections. The last section concludes the paper.2. NAMED ENTITY RECOGNITIONThe standard approach to NER is to use a machine learningtechnique to learn a classifier from a manually annotateddata set.In this setting, a classifier is built that labels each token inthe sentence with a category and segmention label. The cat-egory label indicates the entity type of a word: PER (per-son), LOC (location), ORG (organization) or MISC (mis-cellaneous). The segmention label specifies whether a tokenbegins (B), is inside (I) or outside (O) an entity (hence calledBIO-encoding).Learning algorithms are unable to operate directly on text.First, sentences have to be encoded into a numerical repre-sentation, by defining features that extract signal from text.Features are typically binary predicates that test simple con-ditions (e.g., word at position i equals “postbode”, word atposition i+ 1 contains a digit, and so on).Most of our work went into designing good features. Ourbaseline model mirrors publicly available systems (in par-ticular [1]). It includes word features, shape features, char-acter n-grams (prefixes and suffixes) and a limited set ofconjunctions. Further on, we will describe some extra fea-tures, namely, capitalization and clustering features, thatappeared essential to increase the performance of the base-line system.2.1 Conditional Random FieldsAs a classification algorithm we use linear-chain ConditionalRandom Fields (CRFs) [7], a very popular yet higly com-petitive algorithm. CRFs are discriminative models; theylearn a direct mapping from the input (feature) space tothe output (class label) space, without putting any effortin modeling the input variables’ distribution (as a HiddenMarkov Model would do). This results in a great flexibilityin handling large numbers of arbitrary features, even if theyoverlap and are inter-dependent. As such, virtually any use-ful feature of the input observation can be included into themodel.CRFs are structured classifiers, and are therefore also able todeal with dependencies between output labels. In contrastto many models (such as SVMs and plain logistic regres-sion), CRFs are able to correctly model likely and unlikelylabeling transitions in sentences. Examples of such transi-tions are: ‘successive words are more likely to have the sameentity type’, or ‘new entity types always begin with a B seg-mentation attribute’ (i.e., <O> <I-PER> is not allowed).2.2 Capitalization FeaturesFor the older documents in the VLIB archive, the textualmetadata originate from computer systems that only sup-ported full capitalized input. Capitalization is however avery strong source of information to determine where an en-tity begins, in Dutch and English text. In order to realize adecent performance on such documents, we first tried to re-store capitalization with a sequence tagger. In the end, sim-ple capitalization statistics gathered from Wikipedia provedmuch easier and performed better. For this we define 5 cap-italization classes c: lowercased, capitalized, all uppercased,mixed case and no case. As input features to the CRF wegive the numerical value of the expectations p(c|w)p(c|w) = N(w, c) + λ∑c(N(w, c) + λ)in which N(w, c) is the number of occurrences of word w incapitalization class c and λ is a smoothing constant.2.3 Word ClustersOne of the key challenges of any Natural Language Process-ing (NLP) system is to make robust decisions on new texts,which possibly may deviate from the trainig data. Becauseit will never be possible to gather all words in a training set,the way in which words are represented is crucial.In its most basic form, NLP systems represent words as bi-nary features. For each position in the sentence, only oneof those features is active. This representation fails when anew, out of vocabulary, word has to be encoded. One ap-proach is to introduce a smaller vocabulary of word clusters,grouping similar words into clusters. To obtain such a clus-tering, an unsupervized algorithm is run on as much dataas available (i.e., the whole corpus instead of only the anno-tated data). By optimizing an objective function, each wordis assigned to one cluster. Later, when a word is encounteredthat has never been seen in the training data, the classifiercan make decisions based on the cluster that it belongs to.One of the oldest clustering methods, which is still very com-petitive, is that of brown clusters or class-based n-grams [5].We use predictive exchange clustering (PEC), which is avery fast variant for which also a distributed version ex-ists [8]. PEC works by searching for the optimal clusteringthat maximizes the log likelihood of the input dataL(w; c) =∑ilog p (wi|wi−1) ,=∑ilog p (wi|c (wi)) p (c (wi) |wi−1) . (1)Each term in the above summation consists of a part thatpredicts the current cluster given the previous word, and acomponent that predicts the current word given the currentcluster. Using word frequencies, (1) can be rewritten asfollowsL(w; c) =∑w,cN(w, c) logN(w, c)−∑cN(c) logN(c), (2)with N(w, v) the frequency of bigram (w, v) and N(w, c) thefrequency of word w followed by cluster c.N(w, c) =∑v∈V (c)N(w, v)N(c) =∑w∑v∈V (c)N(w, v)When moving a single word from one cluster to another,only a limited number of terms in (2) are affected, enablingone to evaluate the gain in an efficient manner.The algorithm begins with a random assignment of wordsonto clusters. It then iterates over all words and tries tomove them individually to other classes. Words are movedcluster 48 cluster 52 cluster 69 cluster 76 cluster 96meer hebben twee sint gilberbeter hadden drie brussel laurentminder hopen vier antwerpen cancellaralanger wisten vijf mechelen kadhafihoger vrezen tien leuven vangheluwegroter staken zes hasselt kbcsneller dachten verschillende oostende boussoufasterker mikken zeven kortrijk albertlager verdienden acht aalst binvaker zochten vele turnhout toerismeTable 1: Word clustersonly if it leads to an increase of the objective function. Thisalgorithm terminates when no new moves are found. At thispoint, a local optimum is reached.The resulting clustering is syntactic in nature; words thatfulfill a similar function within the sentence, are placed in thesame cluster. An example is given in Table 1, displaying theten most frequent words in five out of two hundred clusters.Although the table contains some questionable assignments,the NER classifier is supposed to deal with that.2.4 Phrase ClustersAn alternative clustering algorithm was proposed in [6], forclustering phrases using the k-means algorithm. It was ar-gued that the disambiguation power of phrases is strongerthan that of words only. Word clusters have their limita-tions, because words out of their proper context are oftenambiguous. For example, considering the indivual words ofthe phrase “Jan De Nul”, one could easily mistake it for aperson. Looking at the context of this phrase in a large cor-pus, it becomes clear that the phrase actually represents acompany.In a first step, by collecting word occurences in a small win-dow around each phrase, a context vector xphr(w) is cre-ated. The authors of [6] obtained a list of phrases froman anonymized query log. Because we do not have accessto such resources, we first created a list of Named Entitiesas phrases, using an intermediate version of the NER en-gine. All occurrences of these phrases are then looked upover the entire corpus and their surrounding context is ag-gregated, including the occurrences that may have initiallybeen missed by the NER engine. The collected frequencycounts are then rescaled using point-wise mutual informa-tionxphr(w) = log(N(phr, w)∑phr N(phr, w)∑wN(phr, w)).The k-means algorithm creates the final clusters by maxi-mizing the following objective function:E(c) =∑ixi · ck(i)∥∥ck(i)∥∥ .Initially, (i) a number of random phrases are taken as thecluster seeds, (ii) each phrase is assigned to the most similarcenter using cosine similarity, (iii) the centers are recom-puted, and (ii) and (iii) are repeated until no reassignmentscluster 13 cluster 64ronde van catalonie¨ tc de meyltweedaagse van de gaverstreek tc westerloronde van de limousin tc de zwalumhaspengouw tour smash neeroeterenster zlm toer voco tennisclubcluster 94 cluster 120red riding hood dag van de open kerkenmr. popper’s pengiuns wings and wheelsthe hangover 2 landjuweelfestivalrio-3d week van de fair tradesucker punch deistelrockTable 2: Phrase clustersdata set documents tokens termsconll-train 287 207066 25306conll-testa 74 38413 7629conll-testb 119 70071 10917vlib-train 655 41784 8578vlib-test 345 24948 5847mediargus11-train 124 45113 8236mediargus11-test 74 23845 5264Table 3: Data sets to evaluate experimentsoccur. The centers are recomputed by averaging the featurevectors of the cluster elements (thus maximizing the aboveobjective function, given the elements of a clustering).Table 2 lists the best scoring phrases of 4 clusters (out of 128clusters) obtained from running it on the Mediargus data setof 2011, for which a window size of 3 was arbitrarily chosen.3. DATA AND RESULTSIn this section we describe in chronological order the exper-iments we performed. The total number of tokens and thenumber of unique terms within human-annotated data setsare given in Table 3. Although the VLIB and Mediargusdata are not publicly available, these numbers should give agood idea of the size of the annotated data set in comparisonto the publicly available CoNLL data set. The labeled dataset contains about 1000 documents from the VLIB archive,but is equal in size to the second CoNLL test set. The Me-diargus data set is also about the same size.We start with a system trained on VLIB data only. We gainabout 2.5% in F1 measure by adding the CoNLL-2002 datato the training set, which is still far below test accuracy onCoNLL test sets (which is about 78% and 79% respectively).Enabling capitalization features brings us to CoNLL perfor-mance, clearly confirming that capitalization was indeed themain issue. Another 3% gain in performance is obtained byenabling word clusters (from the unlabeled data of VLIBand a complete Wikipedia dump).Testing this version of the NER engine on the Mediargustest set, without further training, resulted in a lower score.After including the Mediargus training set and performingword clustering on the complete collection of news articlesof 2011, performance increased to above that of VLIB. Theinclusion of phrase clusters resulted in our best and finalmodel.To summarize, we list the performance of the final model onall data sets in table 6. These results are obtained by train-ing on CoNLL 2002, VLIB and Mediargus. Word clustersPr Re F1vlib only 83.2 64.3 72.6+CoNLL data 83.9 68.0 75.1+capitalization 82.2 77.5 79.8+word clusters 85.3 80.3 82.7Table 4: Precision, recall and F1-measure for a NERevaluated on the VLIB test setPr Re F1version 1 75.9 77.6 76.7+mediargus11-train 82.8 81.4 82.1+word clusters 85.2 84.0 84.6+phrase clusters 87.0 86.1 86.6Table 5: Precision, recall and F1-measure for a NERevaluated on the Mediargus test setand phrase clusters are derived from Wikipedia, VLIB andMediargus. Although we needed to include extra annotationdata to get to these results (as compared to the publiclyavailable training data), that annotation was needed to getthe NER performing to the same level as publicly availablesystems. The largest increase in performance is due to theword clustering and phrase clustering.4. FUTUREWORKWe were recently granted access to the large newspaperarchive from Mediargus, with news articles from 2000 to2008 (containing 6.3 million articles, as compared to the2011 collection with only 0.5 million articles, used for thispaper). We are hence extending the cluster features for thisentire dataset, to evaluate to what extent these features arecapable of allowing for accurate NER over a large time span.Considering the increase in NER performance based on wordand phrase features for the current test collection, there willmost likely be a significant influence.In the near future we will investigate to what extent it isworthwhile to increase the complexity of the model. Likemost other systems, we use a first order model CRF whichis unable to model long range phenomena and this is visiblein certain aspects of text (e.g., a concatenation of entitiesseparated by commas or a conjunction). Also in the moregeneral case, sequence models are unable to model multiplementions of the same entity in a single document. Ideally,predicting the labels for related mentions should not be doneindependently. Modeling such interactions in a fundamen-tally correct way is a difficult problem and leads to approx-imate methods. Looking at the output of our system, it ishowever an important issue.5. CONCLUSIONSIn this paper, we presented our Named Entity Recognitiontool for Dutch, specifically tailored towards the metadataof a Dutch multimedia archive, but currently also trainedand applied on other textual data such as news collections.Some features of the NER engine, required by the inherentlimitations of the metadata, were explained, along with someof our internal evaluation results during the developmentphase.Pr Re F1conll-testa 86.2 84.2 85.2conll-testb 87.5 85.3 86.4vlib-test 86.4 82.2 84.2mediargus-test 87.0 86.1 86.6Table 6: Performance of final model on all test sets6. ACKNOWLEDGMENTSThis work was in part carried out in the framework of theprojects VLIB and TExSIS, sponsered by the agency forInnovation by Science and Technology (IWT).7. REFERENCES[1] http://www-nlp.stanford.edu/ner/.[2] http://www.mediargus.be/.[3] Terminology Extraction for Semantic Interoperabilityand Standardization (TExSIS).http://www.vlaandereninbeeld.net.[4] Vlaanderen In Beeld (VLIB).http://www.vlaandereninbeeld.net.[5] C. Christodoulopoulos. Two Decades of UnsupervisedPOS induction: How far have we come? Proceedings ofthe 2010 . . . , 2010.[6] D. Lin and X. Wu. Phrase clustering for discriminativelearning. In Proceedings of the Joint Conference of the47th Annual Meeting of the ACL and the 4thInternational Joint Conference on Natural LanguageProcessing of the AFNLP, volume 2, page 1030,Morristown, NJ, USA, 2009. Association forComputational Linguistics.[7] C. Sutton and A. McCallum. An introduction toconditional random fields. Arxiv preprintarXiv:1011.4088, 2010.[8] J. Uszkoreit and T. Brants. Distributed word clusteringfor large scale class-based language modeling inmachine translation. Proceedings of ACL-08: HLT,2008.

Named entity recognition on flemish audio-visual and news-paper archives

https://biblio.ugent.be/publication/2911597/file/2911605

Named entity recognition on flemish audio-visual and news-paper archives

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography