Deleu, Johannes

Demeester, Thomas

Develder, Chris

Sterckx, Lucas

English

Ghent University Academic Bibliography

Using Active Learning and Semantic Clusteringfor Noise Reduction in Distant SupervisionLucas Sterckx, Thomas Demeester, Johannes Deleu, Chris DevelderGhent University - iMindsGhent - Belgiumfirstname.lastname@intec.ugent.beAbstractThe use of external databases to generate training data, also known as Distant Su-pervision, has become an effective way to train supervised relation extractors butthis approach inherently suffers from noise. In this paper we propose a methodfor noise reduction in distantly supervised training data, using a discriminativeclassifier and semantic similarity between the contexts of the training examples.We describe an active learning strategy which exploits hierarchical clustering ofthe candidate training samples. To further improve the effectiveness of this ap-proach, we study the use of several methods for dimensionality reduction of thetraining samples. We find that semantic clustering of training data combined withcluster-based active learning allows filtering the training data, hence facilitatingthe creation of a clean training set for relation extraction, at a reduced manuallabeling cost.1 IntroductionFor the task of extracting relations between entities according to a fixed schema (also known asKnowledge Base Population (KBP)), distantly supervised approaches are currently state-of-the-art [19]. A requisite for the effectiveness of these techniques is the availability of labeled data, whichis expensive to obtain. An approach to solve this issue and produce large quantities of training datais distant supervision (DS) [11]. DS creates labeled data using readily available repositories likeFreeBase or DBpedia with facts like “Person → city-of-residence→ Location” (for the remainderdenoted as “per:city of residence”) and the assumption that every phrase mentioning both entitiesparticipating in the relation expresses the corresponding relation from the database. Using this ap-proach, a large quantity of training data can be generated automatically. However, intuitively thisassumption only holds for a fraction of the extracted mentions, as two entities may co-occur in onesentence for many alternative reasons. A challenge we address in this paper is to develop strategiesto improve the quality of the training data and reduce the amount of noise.In our participation in the Text Analysis Conference for Knowledge Base Population (TAC-KBP)slot filling track organized by NIST [1], a baseline supervised classification using DS was imple-mented as described in [6]. In our submissions, we already showed the value of noise reductionbased on straightforward human annotation of randomly selected training instances: cleaning basedon a classifier (trained on the annotated instances) resulted in 8% higher precision. As this requiredextra manual annotation of the training samples, we search for an efficient way to query the distantlysupervised data and train a classifier using a minimal amount of supervision but an improved noisereduction.This work contributes by presenting a strategy for noise reduction using a supervised classifiertrained using labeled mentions from distantly supervised data. By incorporating semantic relat-edness between the mentions we can use an active learning approach which exploits the resulting1clustering of training data. Intelligent querying of training data clusters and assigning labels to sim-ilar unknown training examples trains a classifier based on less human supervision while optimizingthe capability of separating noisy from true relation contexts.2 Related WorkThe approach of DS was first presented by Mintz et al. [11] for training of binary Support VectorMachines which used a set of lexical and non-lexical features for classification. Since then, severalmethods for noise reduction of the data have been proposed. For a recent survey we refer to Roth etal. [16]. Models like the one proposed by Riedel et al. [14], MultiR [10] and MIML-RE [20] involvelatent variables which model the assumption that at least one generated example for an entity pairand relation is a true positive, or apply a generative model [21]. Our approach is less complex, usinga discriminative classifier based on manually annotated examples of true positive and false positiverelation mentions within each of the generated training sets. Using this classifier, we filter trainingdata explicitly, independent of the entities involved and for each relation separately, solely based onthe surface text.Recent work has combined DS with small amounts of labeled data, these labels are either includeddirectly in a latent variable model [13] or used in an active learning setting. Active learning waspreviously performed in relation extraction by Sun and Grishman [18] for extending a relation ex-traction system to recognize a new type of relation. An approach which uses active learning for DSwas recently proposed by Angeli et al. [2] and successfully applied in the top performing systemin the TAC slot filling competition [19]: they show that a small number of examples can yield im-provements in end-to-end accuracy of the relation extraction using several approaches from activelearning literature and a new metric incorporating uncertainty and representativeness. Our work dif-fers from this and others in that we use a cluster based active learning approach, evaluating directlyon a set of labeled training examples.3 Semantic-Cluster-Aware SamplingOur approach assumes that true positive mentions within each training set are similar to each other,in terms of text and meaning, and tend to cluster together, unlike false positive mentions whichare less similar and more diverse. This inspired us for the application of cluster-aware sampling ofthe training data for the training of the noise-reduction classifier. An active learning approach thatexploits cluster structure in data was introduced by Dasgupta and Hsu [4]. This algorithm takesa pool of unlabeled data and performs hierarchical clustering, resulting in a tree structure of datapoints. The algorithm starts out by randomly querying data points in the vector space and searchesfor a pruning of the tree that optimizes the pureness of each cluster. Each iteration, a number ofdata points are sampled in such a way that less pure clusters are more likely to be sampled from andunseen samples receive the label of the majority of known samples in the cluster it belongs to.As stated in the original paper [4], the algorithm is most effective when pure clusters are foundat the top of the hierarchical tree. Thus, when applied for noise reduction, this approach benefitsfrom relation contexts that are clustered according to the meaning or relation they express. Thesimple bag-of-words representation results in a high dimensionality of the relation contexts withonly few ways of clustering contexts with similar meaning. We need a transformation of the contextsinto a vector space of reduced dimension, with those having a similar expression of relation beingtransformed into similar representations. This is exactly what semantic clustering achieves, i.e.clustering contexts according to meaning.Semantic clustering of relations has been performed on several occasions in the context of Open In-formation Extraction to cluster output having similar meaning and is related to the task of paraphraseand synonym detection [9, 23, 24, 15]. We use a transformation based on a simple composition of thewords participating in the context. While much research has been directed at ways of constructingdistributional representations of individual words, for example co-occurrence based representationsand word embeddings, there has been far less consensus regarding the representation of larger con-structions such as phrases and sentences from these representations. Blacoe et al. [3] show that asimple composition like addition or multiplication of the distributional word representations is com-petitive with more complex operations like the Recursive Neural Networks proposed by Socher et2Figure 1: Visualization of relation con-texts in a semantic vector space for relation“per:spouse of ”.Distantly Supervised Data Active  Learning Noise Filtering Classifier Filtered Distantly Supervised Data True Positive Training Sample for Annotation False Positive Figure 2: Methodology for filtering of noisymentions.al. [17] for detection of paraphrases and synonyms. We chose to ignore word order and sum all dis-tributional representations from words participating in the surrounding context of the mention andnormalize them.4 Experiments and ResultsWe use FreeBase [7] as our source of fact relations by first matching the schema from the TAC-KBPto fields from FreeBase. The participating entities are matched in phrases from articles from theEnglish GigaWord corpus [8]. As part of a participation in the TAC-KBP slot filing competition, ateam of students was asked to assign 2.000 training samples with a True or False label with respectto the 2014 TAC-annotation guidelines for a selection of 12 relations with a large quantity of trainingdata. As these samples were selected at random, some of the relations contained very pure or highlynoisy training sets. Phrases were filtered for duplicates and entity names were removed from thesurface text.Effective representations should be able to separate true examples of a relation being expressedfrom false examples. We visualize this in Figure 1 for the relation per:spouse of. Words in be-tween the subject and object entities of the relation are transformed to their semantic vectors usingword embeddings which are summed and normalized. In our experiments we use the GloVe word-embeddings with 100 dimensions trained on Wikipedia text [12], which are made available from theauthors’ website.1 The resulting sentence representations are clustered and represented in a two-dimensional space using the t-SNE algorithm [22] in Figure 1. True examples of the relation arerepresented in Figure 1 as dark triangles, while false examples are the lighter crosses. The resultingfigure shows that this basic transformation alone is able to capture some of the semantic meaning ofrelations.The active learning strategy is performed on 70% of the DS-data, 30% is set aside to evaluateclassification. The general methodology for filtering distantly supervised data is shown in Figure 2.Previously described active learning iteratively only queries a number of DS-examples, but resultsin a fully labeled distantly supervised data set (each unknown sample then receives the label of themajority of its cluster). The resulting fully labeled DS-data is used to train a logistic regressionclassifier using only word count vectors as features in a basic text classification setting to filterthe training data. At most two words before the entity first mentioned, the words in between theentities and at most two words following the entity mentioned last are included. We compare ourcluster-based active learning in the semantic vector space, with uniform sampling, clustering usingBag-of-Words vectors and clustering after transformation using Latent Semantic Indexing (LSI) [5](also for 100 dimensions). This process is repeated 20 times using stratified cross-folds.1http://nlp.stanford.edu/projects/glove/3l l ll l l l l ll llll l ll l l l l ll lll l ll l ll l ll lll ll l l ll l l l lll ll l l l l ll l lll ll ll ll lllllll ll ll l lll lllllllllllllorg:stateorprovince_of_headquarters org:top_members_employees per:chargesper:age per:spouse per:countries_of_residenceorg:founded_by per:city_of_death0.250.500.751.000.250.500.751.000.250.500.751.000 250 500 750 1000 12500 250 500 750 1000 12500 250 500 750 1000 12500 250 500 750 1000 12500 250 500 750 1000 12500 250 500 750 1000 12500 250 500 750 1000 12500 250 500 750 1000 1250Number of Samples AnnotatedF1−score for Noise Reduction ClassificationlDistant Supervision (Baseline)Uniform SamplingBag−of−Words ClusteringLatent Semantic Indexing ClusteringSemantic Vector Space ClusteringTotal Trainingset SampledFigure 3: Performance of cluster-based active learning approach.Table 1: Macro-average filter performance using 70 labeled distantly supervised training examplesPrecision Recall F1Distant Supervision (Baseline) 51.9 100.0 60.8Random Sampling 72.0 72.8 66.0Bag-of-Words Clustering 73.4 65.2 66.6Latent Semantic Indexing Clustering 73.7 68.5 68.3Semantic Vector Space Clustering 74.6 71.4 71.2Using all of the labeled data, supervised noise reduction is able to increase average fraction of truepositive of the DS training set from 47% to 84% while maintaining a recall of 88%. Noise reductionusing active learning for a selection of relations is presented in Figure 3. Separately for each rela-tion, after each increase of 5% sampled data we calculate the averaged F1 score of the classificationfor each of the strategies. Performance of the noise reduction is highly dependent on the relation.For relatively pure training sets, as is the case for relation “org:state or province of headquarters”,with more than 85% of the training data being positive examples, are hard to filter. For these re-lations supervised filtering appears ineffective or even detrimental, others need a minimum amountof samples to benefit like “per:countries of residence”. Cluster aware active learning is an effec-tive strategy for a majority of the noisy relations, converging faster to the optimal performance offiltering. Overall, performance using semantic clustering of contexts is slightly better than usingLSI clustering, while with very few samples and relations like “per:age” and “per:city of death”performance increase is larger. Another observation is that, because the algorithm also provides thetest samples with a label (based on the majority of the labels in the same cluster as the test sample),classification performance surpasses that of a fully labeled training set while approximately only4half is sampled. Table 1 shows macro average precision, recall and F1 using a minimal amount ofonly 70 samples for noisy relations (fraction of true positives less than 85%).5 ConclusionIn this paper we presented a novel approach for filtering a distantly supervised training set by build-ing a binary classifier to detect true relation mentions, the classifier is trained using a cluster basedactive learning strategy. We show that clustering of relation mentions and adding semantic infor-mation reduces human effort and makes this a promising approach more feasible to filter a widevariety of relations. For future work we suggest the use of more sophisticated methods which takeinto account composition for transforming context to a semantic vector space.References[1] Task Description for English Slot Filling at TAC-KBP. 2014.[2] Gabor Angeli, Julie Tibshirani, JY Wu, and CD Manning. Combining Distant and PartialSupervision for Relation Extraction. Proceedings of the Conference on Empirical Methods inNatural Language Processing, 2014.[3] William Blacoe and Mirella Lapata. A Comparison of Vector-based Representations for Se-mantic Composition. Proceedings of the 2012 Joint Conference on Empirical Methods inNatural Language Processing and Computational Natural Language Learning, 2012.[4] Sanjoy Dasgupta and Daniel Hsu. Hierarchical sampling for active learning. In Proceedingsof the 25th international conference on Machine learning, 2008.[5] Scott C. Deerwester, Susan T Dumais, Thomas K. Landauer, George W. Furnas, and Richard A.Harshman. Indexing by latent semantic analysis. volume 41, pages 391–407, 1990.[6] Matthias Feys, Lucas Sterckx, Laurent Mertens, Johannes Deleu, Thomas Demeester, andChris Develder. Ghent University-IBCN Participation in TAC-KBP 2014 Slot Filling and ColdStart Tasks. 2014.[7] Google. Freebase data dumps. https://developers.google.com/freebase/data, 2014.[8] David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English gigaword. Linguistic DataConsortium, Philadelphia, 2003.[9] Takaaki Hasegawa, Satoshi Sekine, and Ralph Grishman. Discovering relations among namedentities from large corpora. In Proceedings of the 42nd Annual Meeting on Association forComputational Linguistics, 2004.[10] Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S Weld.Knowledge-based weak supervision for information extraction of overlapping relations. InProceedings of the 49th Annual Meeting of the Association for Computational Linguistics:Human Language Technologies-Volume 1, pages 541–550. Association for Computational Lin-guistics, 2011.[11] Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. Distant supervision for relationextraction without labeled data. In Proceedings of the Joint Conference of the 47th AnnualMeeting of the ACL, pages 1003–1011. Association for Computational Linguistics, 2009.[12] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectorsfor word representation. In Proceedings of the Conference on Empirical Methods in NaturalLanguage Processing, 2014.[13] Maria Pershina, Bonan Min, Wei Xu, and Ralph Grishman. Infusion of labeled data into distantsupervision for relation extraction. In Proceedings of the 2014 Conference of the Associationfor Computational Linguistics (ACL 2014), Baltimore, US, June 2014. Association for Com-putational Linguistics.[14] Sebastian Riedel, Limin Yao, and Andrew McCallum. Modeling relations and their mentionswithout labeled text. In Machine Learning and Knowledge Discovery in Databases, pages148–163. Springer Berlin Heidelberg, 2010.5[15] Lorenza Romano, Milen Kouylekov, Idan Szpektor, Ido Dagan, and Alberto Lavelli. Investi-gating a generic paraphrase-based approach for relation extraction. 2006.[16] Benjamin Roth, Tassilo Barth, Michael Wiegand, and Dietrich Klakow. A survey of noisereduction methods for distant supervision. In Proceedings of the 2013 Workshop on AutomatedKnowledge Base Construction, AKBC ’13, pages 73–78, New York, NY, USA, 2013. ACM.[17] Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. Semantic compo-sitionality through recursive matrix-vector spaces. In Proceedings of the 2012 Joint Conferenceon Empirical Methods in Natural Language Processing and Computational Natural LanguageLearning, 2012.[18] Ang Sun and Ralph Grishman. Active learning for relation type extension with local and globaldata views. In Proceedings of the 21st ACM International Conference on Information andKnowledge Management, CIKM ’12, pages 1105–1112, New York, NY, USA, 2012. ACM.[19] Mihai Surdeanu and Heng Ji. Overview of the English Slot Filling Track at the TAC2014Knowledge Base Population Evaluation. 2014.[20] Mihai Surdeanu, Julie Tibshirani, Ramesh Nallapati, and Christopher D Manning. Multi-instance multi-label learning for relation extraction. pages 455–465, 2012.[21] Shingo Takamatsu, Issei Sato, and Hiroshi Nakagawa. Reducing wrong labels in distant super-vision for relation extraction. In Proceedings of the 50th Annual Meeting of the Associationfor Computational Linguistics: Long Papers-Volume 1, pages 721–729. Association for Com-putational Linguistics, 2012.[22] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal ofMachine Learning Research, 2008.[23] Wei Wang, Romaric Besanon, Olivier Ferret, and Brigitte Grau. Semantic clustering of re-lations between named entities. In Adam Przepirkowski and Maciej Ogrodniczuk, editors,Advances in Natural Language Processing, volume 8686 of Lecture Notes in Computer Sci-ence, pages 358–370. Springer International Publishing, 2014.[24] Limin Yao, Aria Haghighi, Sebastian Riedel, and Andrew McCallum. Structured relation dis-covery using generative models. In Proceedings of the 2011 Conference on Empirical Methodsin Natural Language Processing, 2011.6

Using active learning and semantic clustering for noise reduction in distant supervision

https://biblio.ugent.be/publication/5786759/file/5786768.pdf

Using active learning and semantic clustering for noise reduction in distant supervision

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography