de Rijke, M.

Gonzalo, J.

Peetz, M.-H.

Spina, D.

English

International Migration, Integration and Social Cohesion online publications

UvA-DARE is a service provided by the library of the University of Amsterdam (https://dare.uva.nl)UvA-DARE (Digital Academic Repository)Towards an active learning system for company name disambiguation inmicroblog streamsPeetz, M.-H.; Spina, D.; Gonzalo, J.; de Rijke, M.Publication date2013Document VersionFinal published versionPublished inCLEF 2013 : CLEF2013 Working NotesLink to publicationCitation for published version (APA):Peetz, M-H., Spina, D., Gonzalo, J., & de Rijke, M. (2013). Towards an active learning systemfor company name disambiguation in microblog streams. In P. Forner, R. Navigli, D. Tufis, &N. Ferro (Eds.), CLEF 2013 : CLEF2013 Working Notes: Working Notes for CLEF 2013Conference : Valencia, Spain, September 23-26, 2013 (CEUR Workshop Proceedings; Vol.1179). CEUR-WS. http://ceur-ws.org/Vol-1179/CLEF2013wn-RepLab-PeetzEt2013.pdfGeneral rightsIt is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s)and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an opencontent license (like Creative Commons).Disclaimer/Complaints regulationsIf you believe that digital publication of certain material infringes any of your rights or (privacy) interests, pleaselet the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the materialinaccessible and/or remove it from the website. Please Ask the Library: https://uba.uva.nl/en/contact, or a letterto: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. Youwill be contacted as soon as possible.Download date:10 Mar 2023Towards an Active Learning System forCompany Name Disambiguation in MicroblogStreams?Maria-Hendrike Peetz1, Damiano Spina2, Julio Gonzalo2, Maarten de Rijke1{M.H.Peetz,deRijke}@uva.nl,{damiano,julio}@lsi.uned.es1 ISLA, University of Amsterdam2 UNED NLP & IR GroupAbstract. In this paper we describe the collaborative participation ofUvA & UNED at RepLab 2013. We propose an active learning approachfor the filtering subtask, using features based on the detected semanticsin the tweet (using Entity Linking with Wikipedia), as well as tweet-inherent features such as hashtags and usernames. The tweets manuallyinspected during the active learning process is at most 1% of the testdata. While our baseline does not perform well, we can see that activelearning does improve the results.1 IntroductionWith increasing volumes of social media data, social media monitoring and anal-ysis is a vital part of the marketing strategy of businesses. Manual, and increas-ingly also automatic, extraction of topics, reputation, and trends around a brandallows analysts to understand and manage a brand’s reputation. Twitter, in par-ticular, has been used as such a proxy.Efficient manual and automatic extraction requires filtering and disambigua-tion of tweets. Currently, for manual analysis, many non-relevant tweets haveto be discarded. This has an impact on the costs of the analysis. For automaticanalysis, non-relevant tweets might distort the results and decrease reliability.? This research was partially supported by the Spanish Ministry of Education (FPUgrant nr AP2009-0507), the Spanish Ministry of Science and Innovation (HolopediaProject, TIN2010-21128-C02), the Regional Government of Madrid and the ESFunder MA2VICMR (S2009/TIC-1542) the European Community’s Seventh Frame-work Programme (FP7/2007-2013) under grant agreements nr 258191 (PROMISENetwork of Excellence) and 288024 (LiMoSINe project), the Netherlands Organ-isation for Scientific Research (NWO) under project nrs 640.004.802, 727.011.005,612.001.116, HOR-11-10, the Center for Creation, Content and Technology (CCCT),the BILAND project funded by the CLARIN-nl program, the Dutch national pro-gram COMMIT, the ESF Research Network Program ELIAS, the Elite NetworkShifts project funded by the Royal Dutch Academy of Sciences (KNAW), the Nether-lands eScience Center under project number 027.012.105, and the Yahoo! FacultyResearch and Engagement Program.Automatic reliable named-entity disambiguation on social media is therefore anactive field of research. Typically, filtering systems are static (once trained, themodel does not change) and fully automatic (there is no interaction with theanalysts). However, both language and topics around an entity may change overtime and the disambiguation performance is therefore likely to decay. Addition-ally, assuming the improvement in performance are worth it, the time to annotatea handful of tweets a day can easily be spent by analysts. We therefore proposean active learning approach to company name disambiguation. In particular, weanalyze whether the annotation of a small number of tweets (at most 1% of thetest data) per company improves significantly the results.The paper is organized as follows. We continue with an introduction of theproposed approach in Section 2. We proceed with an explanation of the runsin the experimental setup (Section 3) and analyse the results in Section 4. Weconclude in Section 5.2 Proposed ApproachOur proposed approach is based on active learning, a semi-automatic machinelearning process that interacts with the user for updating the classificationmodel. It selects those instances that may maximize the classification perfor-mance with minimal effort. Figure 1 illustrates the pipeline of the system. First,the instances are represented as feature vectors. Second, the instances from thetraining dataset are used for building the initial classification model. Third, thetest instances are automatically classified using the initial model. Fourth, thesystem guesses the candidate to be prompted to the user. This step is performedby uncertainty sampling: the instance with least certain as to be correctly classi-fied is selected. Fifth, the user manually inspects the instance and labels it. Thelabeled instance is then considered to update the model. The active learningprocess is repeated until a termination condition is satisfied (e.g., the number nof iterations performed).1. Feature representation2. Model Training/Update4. Candidate Selection5. User FeedbackTraining DatasetModel3. ClassificationTest Dataset1. Feature representationüûnew labeled instanceFig. 1. Overview of the proposed active learning process.2.1 Feature representationWe tested two feature representations:Bag of Entities + Twitter metadata (BoE). First, an entity linking sys-tem is used to identify relevant Wikipedia articles to a given tweet. The COM-MONNESS probability [5], based on the intra-Wikipedia hyperlinks, is used toselect the most probable entity for each of the longest n-grams that were linkedto Wikipedia articles from corpora related to the specific language. SpanishWikipedia articles are finally translated to the corresponding English Wikipediaarticle by following the interlingual links, using the Wikimedia API.3 Besidesthe entities linked to the tweet, special Twitter metadata—hashtags, usernamesand author of the tweet—is also considered as features.BoE + Bag of Words (BoE+BoW). This second feature representation simplyadds the tokenized text of the tweet to the features in BoE.Features are then weighted by two different weighting functions:4Presence Each term is weighted with binary occurrence in the tweet: 1 ifpresent, 0 otherwise.Pseudo-document TF.IDF As in [10], we consider a pseudo-document Dbuilt from all the tweets given for an entity in the RepLab 2013 training/testdataset and a background corpus C containing all the Di in the RepLab 2013collection. Then, the weight w given to the term t isw(t,D,C) = tf (t,D) · log Ndf (t)where tf (t,D) denotes the term frequency of term t in pseudo-document D anddf (t) denotes the total number of pseudo-documents Di ∈ C in which the termt occurs at least once.2.2 Learning modelWe use Näıve Bayes5 (NB) as a classifier and build an initial model. Our activelearning approach can be split into the selection of candidates for active an-notations, annotation of the candidates and updating the model. Therefore, oneiteration of our learning model follows the following three steps:– Select the best candidate x from the test set T ;3 http://www.mediawiki.org/wiki/API:Properties4 Each linked entity, hashtag, named user and author is treated as a term.5 http://nltk.org– Annotate the candidate x;– Update the model.The annotations are selected from the test set. The test set depends on theexperimental setup: in the cross-validation scenario, we could use the availableannotations, while in the actual testing scenario, we annotated the candidatesmanually.Candidate Selection. Following [8] the candidate selection can be based onuncertainty sampling, margin sampling (in particular for support vector ma-chines [11]) or entropy sampling. Selecting candidates close to the margin ismotivated by selecting candidates where the classification is less confident. Ex-tending this motivation to NB, we choose to look at the probabilities P (C1|F )and P (C2|Fx)) that a candidate x feature vector Fx generates the classes C1 andC2. The candidate x to be annotated from the test set T is:x = arg mini∈T|P (C1 | Fx)− P (C2 | Fx)|. (1)This candidate x is then being annotated and used to update the model.Model updating Due to the speed of the training of the model, we decided toretrain NB with every new instance. We assigned all newly annotated instancesa higher weight than the instances in the original training set.3 Experimental SetupIn the following we describe how we used the training set to select the best featuregroups. Based on this, we describe the runs we submitted. Unlike previous com-pany name disambiguation datasets, such as the WePS-3 ORM dataset [1,12,9]and the RepLab 2012 dataset [2], the RepLab 2013 collection shares the sameset of entities in training and test datasets. As reputation seems to be entity-specific [7], we build models per entity.3.1 Training and parameter selectionSection 2.1 lists two feature representations: bag of entities (BoE) and BoE + bagof words (BoE+BoW). The BoW representation was generated by tokenizing thetext of the tweet using a Twitter-specific tokenizer [6] and removing stopwords(using both Spanish and English stopword lists). Additionally, the feature valuescould be presence or TF.IDF.We used 10 fold cross-validation (10CV) and iterative time-based splitting(ITS) [4] to evaluate the performance of the features. ITS ensures that classi-fication of past tweets cannot be learnt from future tweets. Thus, we sort thetweets according to their time stamps and train the classifier on the first Ktweets and evaluate on the next K tweets. We then train on the first 2K tweetsand evaluate on the next K tweets, etc. The total accuracy is the mean accuracyover all splits. We set K = 10. For both 10CV and ITS we used accuracy as ourevaluation metric.3.2 Submitted runsThe research questions that motivate our selection of submitted runs are:RQ1 Does annotating a small number (15) of tweets from the test set improvethe results?RQ2 Do language-dependent models perform better?We submitted four runs based the research questions, and two additional runsbased on our observation that the data is imbalanced. We submitted two baselineruns without applying active learning: UvA UNED filtering 1 and UvA UNED -filtering 2. The first run is language-dependent, i.e., it uses a different NBmodel per language. The second run is language-independent, i.e., it uses acombined NB model for both languages. In order to answer RQ1, we submittedthe two active learning runs UvA UNED filtering 3 and UvA UNED filtering 4.The initial models are based on UvA UNED filtering 1 and UvA UNED filter-ing 2, respectively. For the language-dependent case, we annotated 10 tweets perentity from the test set for English and 5 tweets per entity for Spanish. In thelanguage-independent case, per entity, we annotated candidate 15 tweets fromthe test set. The runs UvA UNED filtering 5 and UvA UNED filtering 6 areUvA UNED filtering 3 and UvA UNED filtering 4, but when for an entity thetraining set related ratio6 was < 0.1 or > 0.9, we used a winner-takes-all strategy.The winner-takes-all strategy classifies all the tweets as related or unrelateddepending on which class is dominant in the training set.Table 1 provides an overview over the runs. The official results are evaluatedbased on accuracy, reliability (R), sensitivity (S) and F(R,S), the F1-measure ofR and S [3].Table 1. Overview over the runs submitted to RepLab 2013.languagedependent independentinitial models UvA UNED filtering 1 UvA UNED filtering 2active learning (AL) UvA UNED filtering 3 UvA UNED filtering 4AL + winner-takes-all UvA UNED filtering 5 UvA UNED filtering 66 related ratio = |related||related|+|unrelated|.4 ResultsIn the following we analyze the results on the training set in Section 4.1. Wethen elaborate on the official results in Section 4.2.4.1 Preliminary experimentsTable 2 shows the accuracy for the two representation methods BoW+BoE andBoE. It shows results for coding the presence of the feature or the TF.IDF value.We can see that using the TF.IDF coding of the features works better than thePresence encoding. Additionally, we can see that using the BoE alone worksbetter than the combination of both, BoE+BoW.Table 2. Accuracy for the different representation methods tested on the initial model,not split by language.Representation 10CV ITSBoE+BoW - Presence 0.42 0.46BoE+BoW - TF.IDF 0.68 0.72BoE - Presence 0.66 0.70BoE - TF.IDF 0.76 0.79Table 3 shows the accuracy for the two representation methods BoW+BoE andBoE for models built on split languages. It shows results for coding the presenceof the feature or the TF.IDF value. Again, we can see that only using the BoErepresentation performs better in both test settings, 10CV and ITS. Additionally,we can see that here that TF.IDF outperforms performs better than Presence.We therefore choose to use the BoE representation coded with TF.IDF for oursubmitted runs.Table 3. Accuracy for the different represenation methods tested on the initial model,split by language.English SpanishRepresentation method 10CV ITS 10CV ITSBoE + BoW - Presence 0.60 0.60 0.65 0.46BoE + BoW - TF.IDF 0.71 0.72 0.71 0.52BoE - Presence 0.67 0.69 0.74 0.54BoE - TF.IDF 0.78 0.78 0.78 0.574.2 Submitted runsTable 4 shows the results of our official runs with respect to accuracy, reliability(R), sensitivity (S), and F(R,S), the F1 Measure of Reliability and Sensitivity. Wecan see that our baselines as well as the best performing system performs worsethan a simple baseline. The provided baseline selects the class of an instancebased on the class of the closest (using Jaccard similarity) instance in the trainingset.We can, however, see some interesting improvements. For a start, active learn-ing helps. We can see that the use of 1% annotation improves the results for allfour metrics. Secondly, building a language-independent model performs betterthan building two language-dependent models per entity. Finally, we can see thatthe class imbalance also holds in the test set, as assigning the majority class forstrongly skewed data performs much better than using active learning alone.Table 4. Results of the official runs.run id accuracy R S F(R,S)baseline 0.8714 0.4902 0.3200 0.3255UvA UNED filtering 1 0.2785 0.1635 0.1258 0.0730UvA UNED filtering 2 0.2847 0.2050 0.1441 0.0928UvA UNED filtering 3 0.5657 0.2040 0.2369 0.1449UvA UNED filtering 4 0.6360 0.2386 0.2782 0.1857UvA UNED filtering 5 0.7745 0.6486 0.1833 0.1737UvA UNED filtering 6 0.8155 0.6780 0.2187 0.20835 ConclusionsWe have presented an active learning approach to company name disambigua-tion in tweets. For this classification task, we found that active learning doesindeed improve the results in terms of accuracy, reliability, and sensitivity. Sinceour initial models perform significantly lower than an instance-based learningbaseline (probably due to bugs in the implementation), future work will includethe analysis of the impact of active learning on stronger baselines.References1. Amigó, E., Artiles, J., Gonzalo, J., Spina, D., Liu, B., Corujo, A.: WePS-3 Evalua-tion Campaign: Overview of the Online Reputation Management Task. In: CLEF2010 Labs and Workshops Notebook Papers (2010)2. Amigó, E., Corujo, A., Gonzalo, J., Meij, E., de Rijke, M.: Overview of RepLab2012: Evaluating Online Reputation Management Systems. In: CLEF 2012 Labsand Workshop Notebook Papers (2012)3. Amigó, E., Gonzalo, J., Verdejo, F.: A General Evaluation Measure for DocumentOrganization Tasks. In: Proceedings SIGIR 2013 (Jul.)4. Bekkerman, R., Mccallum, A., Huang, G., Others: Automatic Categorization ofEmail into Folders: Benchmark Experiments on Enron and SRI Corpora. Centerfor Intelligent Information Retrieval, Technical Report IR (2004)5. Meij, E., Weerkamp, W., de Rijke, M.: Adding semantics to microblog posts. In:Proceedings of the fifth ACM international conference on Web search and datamining (2012)6. O’Connor, B., Krieger, M., Ahn, D.: Tweetmotif: Exploratory search and topicsummarization for Twitter. Proceedings of ICWSM pp. 2–3 (2010)7. Peetz, M.H., de Rijke, M., Schuth, A.: From sentiment to reputation. In:Forner, P., Karlgren, J., Womser-Hacker, C. (eds.) CLEF (Online WorkingNotes/Labs/Workshop) (2012)8. Settles, B.: Active learning literature survey. Computer Sciences Technical Report1648, University of Wisconsin–Madison (2009)9. Spina, D., Gonzalo, J., Amigó, E.: Discovering filter keywords for company namedisambiguation in Twitter. Expert Systems with Applications 40(12), 4986–5003(2013)10. Spina, D., Meij, E., de Rijke, M., Oghina, A., Bui, M.T., Breuss, M.: Identifyingentity aspects in microblog posts. In: SIGIR. pp. 1089–1090 (2012)11. Tong, S., Koller, D.: Support vector machine active learning with applications totext classification. J. Mach. Learn. Res. 2, 45–66 (Mar 2002)12. Tsagkias, M., Balog, K.: The University of Amsterdam at WePS3. In: CLEF 2010Labs and Workshops Notebook Papers (2010)

Towards an active learning system for company name disambiguation in microblog streams

https://pure.uva.nl/ws/files/55502031/170513_CLEF2013wn_RepLab_PeetzEt2013.pdf

Towards an active learning system for company name disambiguation in microblog streams

Abstract

Similar works

Full text

Available Versions

International Migration, Integration and Social Cohesion online publications