Pivovarova, Lidia

Yangarber, Roman

English

Helsingin yliopiston digitaalinen arkisto

Comparison of Representations of Named Entities for Multi-labelDocument Classification with Convolutional Neural NetworksLidia Pivovarova and Roman YangarberUniversity of Helsinki, FinlandDepartment of Computer Sciencefirst.last@cs.helsinki.fiAbstractWe explore representations for multi-token names in the context of theReuters topic and sector classificationtasks (RCV1). We find that: the best wayto treat names is to split them into tokensand use each token as a separate feature;NEs have more impact on sector classifica-tion than on topic classification; replacingall NEs with special entity-type tokens isnot an effective strategy; representing to-kens by different embeddings for propernames vs. common nouns does not im-prove results. We highlight the improve-ments over state-of-the-art results that ourCNN models yield.1 IntroductionThis paper addresses large-scale multi-class textclassification tasks: categorizing articles in theReuters news corpus (RCV1) according to topicand to industry sectors. A topic is a broad newscategory, e.g., “Economics,” “Sport,” “Health.”A sector defines a narrower business area, e.g.,“Banking,” “Telecommunications,” “Insurance.”We use convolutional neural networks (CNNs),which take word embeddings as input. Typicallyword embeddings are built by treating a corpus asa sequence of tokens, where named entities (NEs)receive no special treatment. Yet NEs may be im-portant features in some classification tasks: com-panies, e.g., are often linked to particular industrysectors, and certain industries are linked to loca-tions. Thus company and location names may beimportant features for sector classification.RCV1 is much smaller than corpora typicallyused to build word embeddings. Thus we utilizeexternal resources—a corpus of approximately 10million business news articles, collected using thePULS news monitoring system (Pivovarova et al.,2013). While nominally RCV1 contains gen-eral news, it is skewed toward business; many ofthe topic labels are business-related (“Markets”,“Commodities”, “Share Capital,” etc.). Thus, weexpect our business corpus to help in learning fea-tures for the Reuters classification tasks.We compare several NE representation to findthe most suitable name features for each task. Weuse the PULS NER system (Grishman et al., 2003;Huttunen et al., 2002a,b) to find NEs and theirtypes—company, location, person, etc. We com-pare various representations of NEs, by buildingembeddings, and training CNNs to find the bestrepresentation. We also compare building embed-dings on the RCV1 corpus vs. using much largerexternal corpora.2 Data and Prior WorkRCV1 (Lewis et al., 2004) is a corpus of about800K Reuters articles from 1996–1997 with man-ually assigned sector and topic labels. Both clas-sifications are multi-label—each document mayhave zero or more labels. While all documentshave topic labels, only 350K have sector labels.While RCV1 appears frequently in publishedresearch, few authors tackle the full-scale classi-fication problem. Typically they use subsets of thedata: (Daniely et al., 2017; Duchi et al., 2011) useonly the four most general topic labels; (Dredzeet al., 2008) use 6 sector categories to explore bi-nary classification, (Daniels and Metaxas, 2017)use a subset of 6K articles. Even when the entiredataset is used, the training-text split varies acrosspapers, because the “original” split (Lewis et al.,2004) is impractical for most purposes: 23K in-stances for training, and 780K for testing.Another problem that complicates comparisonis the lack of consistency in evaluation metrics  A US appeals Court  revived a civil suit accusing Apple of creating a monopoly text representationwith word embeddingsfirst convolutional layerwith multiple filter widthsfeature maps feature mapssecond convolutional layerwith multiple filter widthsmax-pooling fully-connected layerwith dropout and sigmoid outputFigure 1: Architecture of the convolutional neural networksused to evaluate classifier performance. The mostcommon measures for multi-class classificationare macro- and micro-averaged F-measure, whichwe use in this paper. However, others use othermetrics. For example, (Liu et al., 2017) use pre-cision and cumulative gain at top K—measuresadopted from information retrieval. This is notcomparable with other work, because these met-rics are used not only to report results, but alsoto optimize the algorithms during training. Thenotion of the best classifier differs depending onwhich evaluation measure is used. Thus, althoughRCV1 is frequently used, we find few papers di-rectly comparable to our research, in the sense thatthey use the entire RCV1 dataset and report micro-and macro-averaged F-measure.To the best of our knowledge, our previouswork (Du et al., 2015) was the only study of theutility of NEs for RCV1 classification. We demon-strated that using a combination of keyword-basedand NE-based classifiers works better than eitherclassifier alone. In that paper we applied a rule-based approach for NEs, and did not use NEs asfeatures for machine learning.3 ModelThe architecture of our CNN is shown in Figure 1.The inputs are fed into the network as zero-paddedtext fragments of fixed size, with each word repre-sented by a fixed-dimensional embedding vector.The inputs are fed into a layer of convolutional fil-ters with multiple widths, optionally followed bydeeper convolutional layers. The results of the lastconvolutional layer are max-pooled, producing avector with one scalar per filter. This is fed into afully-connected layer with dropout regularization,with one sigmoid node in the output layer for eachof the class labels. For each class label, a cross-entropy loss is computed. Losses are averagedacross labels, and the gradient of the loss is back-propagated to update the weights. This is similarto the model (Kim, 2014) used for sentiment anal-ysis. The key differences are that our model usesan arbitrary number of convolutions, and that weuse sigmoid rather than softmax on output, sincethe labels are not mutually exclusive.To train the model we used a random split: 80%of the data used for training, 10% developmentset, and 10% test set. The development set is usedto determine when to stop training, and to tune aset of optimal thresholds {θi} for each label i—ifthe output probability pi is higher than θi, the la-bel is assigned to the instance, otherwise it is not.To find the optimal threshold, we optimize the F-measure for each label. The test set is used to ob-tain the final, reported performance scores.Our focus is this paper is data representation,thus we defer the tuning of hyper-parameters forfuture work. All experiments use the same net-work structure: 3 convolution layers with filtersizes {3,7,11}, {3,7,11}, and {3,11}, with 512,256 and 256 filters of each size, respectively. Theruns differ only in the input embeddings they use.4 Data RepresentationWe train the embeddings using GloVe (Penningtonet al., 2014). As features we use lower-cased lem-mas of all words. The rationale for this is that ourcorpora are relatively small, so the data are sparseand not sufficient to build embeddings from sur-face forms. We tune the embeddings while train-ing the CNN, updating them at each iteration.We explore several name representations, usingour NER system:• type: each entity is represented by a spe-cial token denoting its type—C-company, C-person, C-location, etc, and C-name if thetype is not determined. The model learns oneembedding for each of these tokens.• name: each name gets its own embedding;multi-word names treated as a single token.• split-name: multi-word names are split intotokens, and each token has its own embed-ding; the motivation is that some companynames may contain informative parts—e.g.,Air Baltic, Delta Airlines—which may indi-cate that these companies operate in the samefield; these name parts may be more usefulthan the name as a whole.• split-name+common: similar to the above,but tokens inside names and in common con-text are distinguished; the motivation is thatsome words may be used in names with-out any relation to the company’s line ofbusiness—e.g., Apple, Blackberry—and theirusage inside names should not be mixed withtheir usage as common nouns.In the experiments, we build GloVe embeddingsfrom two corpora: RCV1 only, and RCV1 plusour external corpus. For comparison, we also use200-dimensional embeddings trained on a 6 bil-lion general corpus (glove-6B), provided by theGloVe project.0 This corresponds to our split-name representation mode.To illustrate the effect of the different tokenrepresentations, Table 1 shows ten words near-est to the sample lemmas: apple and airline.When name representation is used, the tokenapple is ambiguous, its nearest neighbors areboth fruit words (pear) and computer words (ap-ple computer). In type representation, the “com-puter” meaning disappears, since all mentions ofApple as company are represented by the specialtoken C-company. When using glove-6B, the fruitmeaning is absent, and all neighbors are computer-related words. The token airline does not exhibitsuch ambiguity, and all representations producesimilar nearest neighbors.0https://nlp.stanford.edu/projects/glove/applename type glove-6Bpear pear iphoneunpasteurized unpasteurized microsoftjuice juice intelapple computer fruit macintoshodwalla salmonella ipodstrawberry peach ibmfruit taint ipadmacintosh orange softwaremeat crate googlepear board strawberry itunesairlinename type glove-6Bcarrier carrier airlinesflight flight airwaysbritish airways passenger lufthansaamerican airlines aircraft carrierair france airport flightspassenger air flightlufthansa pilot pilotsair route qantasunited airlines plane alitaliaaircraft aviation klmsplit-name+commonapple apple NE airline airline NEpear computer NE airlines NE malaysian NEjuice macintosh NE airways NE scandinavian NEunpasteurized amelio NE carrier airlines NEodwalla NE operating-system flight system NEfruit compaq NE air NE pilots NEanthrax microsoft NE passenger air NEsalmonella oracle NE lufthansa NE klm NErotten ibm NE pilot passengers NEunpasteurised software aircraft jet NEstrawberry jobs NE route tajudin NETable 1: Nearest neighbors for sample words usingvarious word representations.In the split-name+common representationmode, each lemma may produce two vectors, onefor a common noun and one for a proper noun(inside a name). As the table shows, apple as acommon noun has a clear “fruit” meaning; the onecompany appearing among the neighbors is a juiceproducer, Odwalla. The nearest neighbors for ap-ple NE, in name context, include IT companies.The tokens airline and airline NE have no clearsemantic distinction, with similar nearest neigh-bors. In such cases there is no clear advantage inusing two embeddings rather than one.We test all of the above name representationsexperimentally, to determine which is more usefulin the document classification tasks.5 Results and DiscussionExperimental results are presented in Tables 2and 3. We compare our results with those foundin related work, described in Section 2, focusingon micro- and macro-averaged F-measure—µ-F1and M-F1, respectively. The experimental settingsdiffer in the various papers, which makes precisecomparison difficult. For example, several pre-vious papers use the “standard split,” (proposedin (Lewis et al., 2004)), which contains only 23KAlgorithm (prior) M-F1 µ-F1SVM (Lewis et al., 2004) 29.7 51.3SVM (Zhuang et al., 2005) 30.1 52.0Naive Bayes (Puurula, 2012) — 70.5Bloom Filters (Cisse et al., 2013) 47.8 72.4SVM + NEs (Du et al., 2015) 57.7 63.8RCV1 embeddingsCNN type 32.2 58.4CNN name 61.0 80.2CNN split-name 63.6 82.0CNN split-name+common 44.3 68.3RCV1 + external corpusCNN type 47.7 72.6CNN name 55.2 78.4CNN split-name 60.7 80.3CNN split-name+common 38.0 66.0CNN split-name (Glove-6B) 55.7 78.4Table 2: Sector classification results on RCV1.training instances, which is not sufficient for learn-ing word embeddings.Compared to the reported state-of-the-art resultson Sector Classification (Table 2), our best modelyields a 10% gain in µ-F1, (Cisse et al., 2013), anda 6% gain in M-F1 (Du et al., 2015). The best µ-F1and M-F1 results are obtained by the same model.1On Topic Classification (Table 3), our µ-F1 re-sults show a modest improvement of 0.5% in F-measure—or a 3.5% (averaged) error reduction—over state of the art (Johnson and Zhang, 2015).2As seen in Table 2, the best data representa-tion for Sector Classification, is split-name, whereeach token has the same embedding regardlesswhether it is used in a proper-name or a common-noun context. The worst performing name repre-sentation is type, where names are mapped to spe-cial “concepts” (C-company, C-person etc.), andeach concept has its own embedding. This in-dicates the importance of the tokens inside thenamed entities for Sector Classification, and sup-ports the notion that company names mentioned intext correlate with sector labels.Results for Topic Classification are in Table 3.The best data representation is again split-name,though the difference between representations isless pronounced than in the case of Sector Classi-fication, and using type does not lead to a signif-icant drop in model performance. This suggeststhat proper names are less important for Topic1In prior work, state of the art was achieved by differentmodels.2Interestingly, the best result for M-F1 on Topics is stillin prior work: i.e., these prior models perform better on veryinfrequent topics. This is to be explored in future work.Algorithm (prior) M-F1 µ-F1SVM (Lewis et al., 2004) 61.9 81.6ANN (Nam et al., 2014) 69.2 85.3CNN (Johnson and Zhang, 2015) 67.1 85.7RCV1 embeddingsCNN type 65.5 85.5CNN name 66.7 86.2CNN split-name 66.5 86.2CNN split-name+common 66.6 86.2RCV1 + external corpusCNN type 64.9 85.6CNN name 66.4 86.2CNN split-name 65.7 85.9CNN split-name+common 65.6 85.8CNN split-name (Glove-6B) 65.8 85.8Table 3: Topic classification results on RCV1.(event) classification, and supports the intuitionthat entity names (e.g., companies) are less corre-lated with the types of events in which the entitiesparticipate in business news. However, there maybe correlations between industry sectors and top-ics/events: e.g., mining or petroleum companiesrarely launch new products. This may explain whythe split-name representation appears to be betterfor Topic Classification. One possible next step isto build CNNs that jointly model Topics and Sec-tors; we plan to explore this in future work.Surprisingly, using external corpora did not im-prove the models’ performance, as indicated byboth Sector and Topic results (Tables 2 and 3, re-spectively). This may mean that the genre and thetime period of the news corpus are more relevantfor building embeddings than the size of the cor-pora. However, other factors may contribute aswell, e.g., our hyper-parameter combination maynot be optimal for these embeddings. Neverthe-less, the results follow the same pattern: the bestname representation is split-name and the differ-ence between representations is more pronouncedfor Sector than for Topic classification.In conclusion, our contribution is two-fold. Onone classic large-scale classification task, sectors,our proposed CNNs yield substantial improve-ments over state-of-the-art; on topics—a modestimprovement in µ-F-measure. Further, to the bestof our knowledge, this is the first attempt at a sys-tematic comparison of NE representation for textclassification. More effective ways of representingNEs should be explored in future work, given theirimportance for the classification tasks, as demon-strated by the experiments we present in this paper.ReferencesMoustapha M. Cisse, Nicolas Usunier, Thierry Arti,and Patrick Gallinari. 2013. Robust Bloom filtersfor large multilabel classification tasks. In Advancesin Neural Information Processing Systems, pages1851–1859.Zachary Alan Daniels and Dimitris N Metaxas. 2017.Addressing imbalance in multi-label classificationusing structured hellinger forests. In AAAI, pages1826–1832.Amit Daniely, Nevena Lazic, Yoram Singer, and KunalTalwar. 2017. Short and deep: Sketching and neuralnetworks.Mark Dredze, Koby Crammer, and Fernando Pereira.2008. Confidence-weighted linear classification. InProceedings of the 25th International Conference onMachine Learning, ICML ’08, pages 264–271, NewYork, NY, USA. ACM.Mian Du, Matthew Pierce, Lidia Pivovarova, and Ro-man Yangarber. 2015. Improving supervised clas-sification using information extraction. In Interna-tional Conference on Applications of Natural Lan-guage to Information Systems, pages 3–18. Springer.John Duchi, Elad Hazan, and Yoram Singer. 2011.Adaptive subgradient methods for online learningand stochastic optimization. J. Mach. Learn. Res.,12:2121–2159.Ralph Grishman, Silja Huttunen, and Roman Yangar-ber. 2003. Information extraction for enhanced ac-cess to disease outbreak reports. Journal of Biomed-ical Informatics, 35(4):236–246.Silja Huttunen, Roman Yangarber, and Ralph Grish-man. 2002a. Complexity of event structure in IEscenarios. In Proceedings of the 19th InternationalConference on Computational Linguistics (COLING2002), Taipei.Silja Huttunen, Roman Yangarber, and Ralph Grish-man. 2002b. Diversity of scenarios in informationextraction. In Proceedings of the Third InternationalConference on Language Resources and Evaluation(LREC 2002), Las Palmas de Gran Canaria, Spain.Rie Johnson and Tong Zhang. 2015. Semi-supervisedconvolutional neural networks for text categoriza-tion via region embedding. In Advances in neuralinformation processing systems, pages 919–927.Yoon Kim. 2014. Convolutional neural networks forsentence classification. In Proceedings of the Con-ference on Empirical Methods in Natural LanguageProcessing (EMNLP).David D. Lewis, Yiming Yang, Tony G. Rose, and FanLi. 2004. RCV1: A new benchmark collection fortext categorization research. The Journal of Ma-chine Learning Research, 5:361–397.Jingzhou Liu, Wei-Cheng Chang, Yuexin Wu, andYiming Yang. 2017. Deep learning for extrememulti-label text classification. In Proceedings of the40th International ACM SIGIR Conference on Re-search and Development in Information Retrieval,pages 115–124. ACM.Jinseok Nam, Jungi Kim, Eneldo Loza Mencı́a, IrynaGurevych, and Johannes Fürnkranz. 2014. Large-scale multi-label text classification—revisiting neu-ral networks. In Joint european conference onmachine learning and knowledge discovery indatabases, pages 437–452. Springer.Jeffrey Pennington, Richard Socher, and Christo-pher D. Manning. 2014. GloVe: Global vectors forword representation. In Empirical Methods in Nat-ural Language Processing (EMNLP), pages 1532–1543.Lidia Pivovarova, Silja Huttunen, and Roman Yangar-ber. 2013. Event representation across genre. InProceedins of the 1st Workshop on Events: Defi-nition, Detection, Coreference, and Representation,NAACL HLT.Antti Puurula. 2012. Scalable text classification withsparse generative modeling. In Patricia Anthony,Mitsuru Ishizuka, and Dickson Lukose, editors, PRI-CAI 2012: Trends in Artificial Intelligence, volume7458 of Lecture Notes in Computer Science, pages458–469. Springer Berlin Heidelberg.Dong Zhuang, Benyu Zhang, Qiang Yang, Jun Yan,Zheng Chen, and Ying Chen. 2005. Efficient textclassification by weighted proximal SVM. In FifthIEEE International Conference on Data Mining.

Comparison of Representations of Named Entities for Multi-label Document Classification with Convolutional Neural Networks

https://helda.helsinki.fi/bitstream/10138/240080/4/2018_REPL_ACL_reuters_short.pdf

Comparison of Representations of Named Entities for Multi-label Document Classification with Convolutional Neural Networks

Abstract

Similar works

Full text

Available Versions

Helsingin yliopiston digitaalinen arkisto