We introduce a new approach for training named-entity pair embeddings to improve relation extraction performance in the biomedical domain. These embeddings are trained in
 an unsupervised manner, based on the principles of distributional 
semantics. By adding them to neural network architectures, we show that 
improved F-Scores are achieved. Our best performing neural model which 
utilizes entity-pair embeddings along with a pre-trained BERT encoder, achieves an F-score of 77.19 on CHEMPROT (Chemical-Protein) relation extraction corpus, setting a new state-of-the-art result for the task.</p

Ginter F.

Mehryary F.

Moen H.

Salakoski T.

UTUPub

   This is a self-archived – parallel-published version of an original article. This version may differ from the original in pagination and typographic details. When using please cite the original.  AUTHOR Farrokh Mehryary, Hans Moen, Tapio Salakoski, Filip Ginter TITLE   Entity-Pair Embeddings for Improving Relation Extraction in the Biomedical Domain YEAR  2020 VERSION  Publisher´s PDF CITATION Farrokh Mehryary, Hans Moen, Tapio Salakoski, Filip Ginter: Entity-Pair Embeddings   for Improving Relation Extraction in the Biomedical Domain. ESANN 2020    proceedings, European Symposium on Artificial Neural Networks, Computational   Intelligence and Machine Learning. Online event, 2-4 October 2020, i6doc.com publ.,   ISBN 978-2-87587-074-2. Available from http://www.i6doc.com/en/.   Entity-Pair Embeddings for Improving RelationExtraction in the Biomedical DomainFarrokh Mehryary1,2, Hans Moen1, Tapio Salakoski1, Filip Ginter11- Turku NLP Group, Department of Future Technologies,University of Turku, Turku, Finland2- University of Turku Graduate School (UTUGS), Turku, FinlandAbstract. We introduce a new approach for training named-entity pairembeddings to improve relation extraction performance in the biomedicaldomain. These embeddings are trained in an unsupervised manner, basedon the principles of distributional semantics. By adding them to neu-ral network architectures, we show that improved F-Scores are achieved.Our best performing neural model which utilizes entity-pair embeddingsalong with a pre-trained BERT encoder, achieves an F-score of 77.19 onCHEMPROT (Chemical-Protein) relation extraction corpus, setting a newstate-of-the-art result for the task.1 IntroductionThe significant amount and the increasing publication rate in the biomedicaldomain make it difficult for biomedical researchers to acquire and maintain allinformation that is necessary for their research. Biomedical relation extrac-tion systems aim to address this problem. These systems can periodically scanthe whole publicly available literature (PubMed article abstracts and PubMedCentral Open Access (PMCOA) full article texts) and extract relations andinteractions of biomedical named entities from the texts and build up-to-daterelation databases or molecular interaction networks to facilitate biomedical re-search. A number of shared task challenges have been organized to promote thedevelopment and evaluation of such systems. For example, the BioCreative VIshared task [1] was recently organized, which provided the CHEMPROT cor-pus for chemical-protein relations extraction. Since the CHEMPROT corpus isfairly new, relatively large and carefully annotated, it has become an importantbenchmark for evaluating modern relation extraction systems.So far the best results for relation extraction have been achieved using systemensemble approaches. On the CHEMPROT corpus, the best result during theBioCreative VI challenge was obtained by Peng et al. [2] (an F-score of 64.10)by using an ensemble system consisting of a recurrent neural network system,a convolutional neural network system and a support vector machine system.Recently, the introduction of transformer-based language representation mod-els such as BERT [3], impacted the field and resulted in unprecedented jumpsin F-score on many data sets. The state-of-the-art result on CHEMPROT isrecently achieved by Lee et al. [4] (an F-score of 76.46), by pre-training theBERT encoder on PubMed sentences and fine-tuning it with a decision layeron the CHEMPROT data for relation extraction. This 12 percentage pointsESANN 2020 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Online event, 2-4 October 2020, i6doc.com publ., ISBN 978-2-87587-074-2. Available from http://www.i6doc.com/en/.613increase in F-score is substantial and raises the question to what extent furtherimprovements on top of BERT can be achieved.Biomedical literature includes a lot of information about the relations andinteractions of biomedical named entities (e.g. genes, proteins, chemicals, anddrugs). We aim to leverage this literature-wide information using unsupervisedmethods and for every unique named-entity pair (Ei, Ej), capture all stated in-formation about Ei and Ej and their relations and build embeddings (vectorrepresentations) of entities and entity pairs. Similarly to word2vec [5] for ordi-nary words, our objective is for similar proteins, chemicals and protein-chemicalpairs to obtain similar embeddings. We are especially interested in investigat-ing the possible effects of incorporating these entity and entity-pair embeddingsinto neural models, in order to improve the performance in relation extractiontasks in the biomedical domain. Given that manually annotated training datafor such tasks is usually limited, the domain is an obvious target for transferlearning through pre-trained embeddings. We hypothesize that by pre-trainingand incorporating the entity-pair embeddings into neural networks, we can im-prove the performance in relation extraction tasks at hand, potentially betterthan using individual entity embeddings alone.In this paper, we explore different approaches for pre-training vector repre-sentations for biomedical entities and entity pairs. We concentrate on the chem-ical and protein named entities in the CHEMPROT corpus and train differenttypes of entity and pair embeddings. We show that when these embeddings areadded into BERT-based neural architectures, they can boost the performance ofrelation extraction. Our approaches are inspired by the work of Levy and Gold-berg [6], using richer contexts to extend the skip-gram architecture of word2vecmodel introduced by Mikolov et al. [5].2 MethodWe propose to pre-train embeddings for named entities and named-entity pairsusing a word2vec skip-gram style training, whereby the named entities, or entitypairs are given as the focus terms, and elements from their contexts are predicted.We will investigate two ways to define the context: a simple linear context ofwords as in the base word2vec, and as an alternative a rich set of featuresextracted from the context. These features have previously been shown to beuseful in supervised relation extraction and one might therefore expect theyresult in embeddings informative for relation extraction. More specifically, wewill rely on the Turku Event Extraction System (TEES) [7] to generate thesefeatures, a system that has achieved numerous top ranks in biomedical relationextraction tasks.2.1 Entity and entity-pair embeddings pre-trainingWe obtain the list of all chemical-protein pairs in the CHEMPROT corpus andfind all sentences in PubMed and PMCOA texts [8] that contain at least one pair.For simplicity, we use exact matching approach when searching for the entitiesESANN 2020 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Online event, 2-4 October 2020, i6doc.com publ., ISBN 978-2-87587-074-2. Available from http://www.i6doc.com/en/.614in the texts. We then extract a set of features for each pair using the TEES sys-tem, including (1) word/lemma/POS-tag and dependency-type N-grams alongthe shortest path connecting the two entities in the sentence dependency parsegraph, (2) word/lemma/part-of-speech N-grams of the words that are locatedwithin [-3,+3] words of the two entities, and (3) type and location of all biomed-ical entities occurring in the sentence. We use the word2vecf toolkit [6] for train-ing the embeddings and use either surrounding words as the context or TEESfeatures. Since simultaneous training of embeddings for entities and entity pairscan impact the final model (due to the shared output layer in the word2vecmodel), we train separate embedding models that include only entity pairs, onlyentities, or both pairs and entities, resulting in six models (see Table 1).Model Content Context for trainingP TEES only pair embeddings TEES featuresP Words only pair embeddingsunion of the words surrounding thetwo entitiesE TEES only entity embeddingsunion of TEES features for everypair that includes the entityE Words only entity embeddings words surrounding the entityPE TEES pair and entity embeddings TEES featuresPE Words pair and entity embeddings surrounding wordsTable 1: Description of the different embedding models.2.2 Relation extraction with entity and entity-pair embeddingsWe incorporate the pre-trained embeddings into the following neural networkarchitectures: (1) BERT MASK: this architecture is developed by Lee et al. [4] andhas achieved the state-of-the-art on CHEMPROT corpus. A BERT encoderpre-trained on PubMed sentences is fine-tuned on the CHEMPROT training setwith a decision layer for relation extraction task. This layer predicts one of thefive possible relation types between the two entities, or a negative label for norelation. We replicate the method as well as use the pre-trained BERT modelof Lee et al. [4]. In the BERT MASK method, the entities are replaced with pre-defined tags (e.g. @PROTEIN$) to inform the classifier where the two entitiesare located in a sentence; (2) BERT MARK: the BERT MASK model hides all infor-mation about the two entities in the sentence as a consequence of its maskingstrategy. Since the entity and pair embedding vectors we pre-train provide infor-mation about the entities, an improvement on top of the BERT MASK model mightbe due to this fact. Therefore, as a fairer baseline, we introduce the BERT MARKmodel (identical to the BERT MASK) except we mark the two entities using thespecial “unused” symbols in BERT vocabulary1 (e.g. [unused1]17β-estradiolbenzoate[unused2]); (3) BERT+Pairs: this model is similar to the BERT MARK1This provided better results compared to using normal characters to mark entity spans(which was used by Lee et al. [4]) since the pre-trained BERT has no notion of the unusedsymbols.ESANN 2020 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Online event, 2-4 October 2020, i6doc.com publ., ISBN 978-2-87587-074-2. Available from http://www.i6doc.com/en/.615model, except the pair embedding vector is concatenated to the BERT sentencerepresentation vector ([CLS] token), transformed through a 1024-dimensionaldense layer with tanh activation, and then presented to the decision layer. Thedense layer with the non-linear activation function learns to combine BERT fea-tures with the pair embedding features; (4) BERT+Entities: this model is similarto the BERT+Pairs model, except we concatenate the chemical and the proteinembedding vectors (not the pair vector); (5) BERT+Pairs+Entities: This modelis similar to previous models, except we concatenate chemical, protein, and pairvectors. In all models, we use the exact hyper-parameters used by Lee et al. [4]and optimize the learning-rate by grid search on the development set.3 Evaluation and resultsWe evaluate all approaches on the CHEMPROT corpus which contains 4,157training examples, 2,416 examples in the development set and 3,458 examplesin the test set. Chemical-protein pairs can have one of 5 positive relations (e.gup regulation) or no interaction at all (negative). We use the official evaluationscript provided by the task organizers which calculates the micro-averaged F-score of the positive classes as the task metric. Since initial random weights ofa neural model can slightly impact the final F-score, we repeat each experiment(training on the training set and predicting development or test set) for 10times which results in obtaining 10 F-scores for each approach. We report theaverage and standard deviation of the F-scores. We use the two-tailed two-sample independent t-test (Welch’s t-test) to establish statistical significance.3.1 Model selectionTable 2 summarizes the results on the development set, reporting statistical sig-nificance at p = 0.1. BERT MARK outperforms BERT MASK, suggesting that markingshould be preferred over the masking approach. In fact, based on column G1, allmodels that utilize marking (rows 2-9) outperform the approach of Lee et al. [4].However, as discussed previously, for us BERT MARK is considered the baseline andas column G2 shows, the models that used only entity embeddings (rows 3,4),do not achieve statistically better results than the baseline. Similarly, the modelthat used pairs (trained with words contexts, row 5) does not achieve a betterresult, in contrast to the model that used pair embeddings (trained with TEEScontext, row 6). All models that utilized pre-trained entity and pair embeddings(rows 7,8) achieve statistically better results than the baseline. We further con-duct another experiment and test the effect of randomly initializing entity andpair embeddings instead of using pre-trained embeddings, to check if the neuralmodel can efficiently learn these embeddings from scratch. However, this modelis not able to outperform the baseline (row 9). Thus we conclude pre-trainingembeddings on the literature is indeed useful. Based on these development setresults, only 3 approaches outperform the baseline (rows 6-8).ESANN 2020 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Online event, 2-4 October 2020, i6doc.com publ., ISBN 978-2-87587-074-2. Available from http://www.i6doc.com/en/.616# Neural modelEmbed-dingsmodelF-score(mean)F-score(std)G1 G21 BERT MASK - 78.41 0.53 - Yes2 BERT MARK - 78.96 0.41 Yes -3 BERT+Entities E Words 79.23 0.43 Yes No4 BERT+Entities E TEES 79.15 0.42 Yes No5 BERT+Pairs P Words 79.27 0.43 Yes No6 BERT+Pairs P TEES 79.36 0.32 Yes Yes7 BERT+Pairs+Entities PE Words 79.47 0.37 Yes Yes8 BERT+Pairs+Entities PE TEES 79.55 0.40 Yes Yes9 BERT+Pairs+EntitiesRandomlyinitialized79.05 0.46 Yes NoTable 2: Results on CHEMPROT development set. Columns G1 and G2 show if basedon the statistical test, the F-score mean is significantly different from the F-score meanof BERT MASK and BERT MARK models respectively.3.2 Final evaluationWe compare our best models selected on the development set (rows 6-8 in Ta-ble 2) with the best previous result of Lee et al. [4] (an F-score of 76.46) onthe test set. To assess the statistical significance, we use the one-sample t-test(p = 0.05). We also evaluate the BERT MASK model to check how well we havebeen able to replicate the method of Lee et al. [4].# Neural modelEmbed-dingsmodelF-score(mean)F-score(std)G1Lee et al. [4] - 76.46 - -1 BERT MASK - 76.41 0.72 No2 BERT+Pairs P TEES 77.13 0.53 Yes3 BERT+Pairs+Entities PE Words 76.71 0.77 No4 BERT+Pairs+Entities PE TEES 77.19 0.49 YesTable 3: Results on CHEMPROT test set. Column G1 shows if based on the statisticaltest, F-score mean is significantly different from the F-score of Lee et al. [4].The test set results (Table 3) validate our replication of the Lee et al. method(row 1). The model that uses surrounding words as the context (row 3) doesnot outperform the baseline (at p = 0.05), however the models that use TEESfeatures (rows 2,4), outperform the best previous result, suggesting that pair-embeddings with rich feature-based context can improve upon a strong BERT-based baseline. Our best model (row 4) sets a new state-of-the-art for the task,improving the best previous score by 0.73 percentage points.ESANN 2020 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Online event, 2-4 October 2020, i6doc.com publ., ISBN 978-2-87587-074-2. Available from http://www.i6doc.com/en/.6174 Conclusion and future workWe compared different approaches for pre-training entity and entity-pair embed-dings to improve relation extraction performance in the biomedical domain. Wehave shown that (1) incorporation of these embeddings into neural models helpsin achieving better performance, (2) using rich features as context (instead ofusing the surrounding words, i.e. the normal word2vec approach) leads to betterresults; (3) using pair embeddings with/without entity embeddings leads to bet-ter results compared to using entity embeddings alone. Our best model achievesan F-score of 77.19, improving the best previous result by +0.73pp over a strongbaseline, and setting a new state-of-the-art for the task. As future work, we aimto investigate the effect of entity and entity-pair embeddings on other biomedicalrelation extraction data sets.5 AcknowledgementsWe would like to thank Dr. Sampo Pyysalo for his invaluable recommendationsand Dr. Jari Björne for his help in running TEES software. The research waspartly funded by the Finnish Cultural Foundation and Academy of Finland(315376).References[1] Martin Krallinger, Obdulia Rabal, Saber A Akhondi, Mart́ın Pérez-Pérez, Jesus Santa-maŕıa, et al. Overview of the BioCreative VI chemical-protein interaction track. In Pro-ceedings of the sixth BioCreative challenge evaluation workshop, volume 1, pages 141–146,Bethesda, MD, USA, 2017.[2] Yifan Peng, Anthony Rios, Ramakanth Kavuluru, and Zhiyong Lu. Extracting chemical-protein relations with ensembles of SVM and deep learning models. Database, 2018, 072018.[3] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-trainingof deep bidirectional transformers for language understanding. In Proceedings of the 2019Conference of the North American Chapter of the Association for Computational Linguis-tics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186,Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.[4] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, et al. BioBERT:a pre-trained biomedical language representation model for biomedical text mining. Bioin-formatics, 09 2019.[5] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeffrey Dean. Distributedrepresentations of words and phrases and their compositionality. In Advances in NeuralInformation Processing Systems 26, pages 3111–3119. Curran Associates Inc., 2013.[6] Omer Levy and Yoav Goldberg. Dependency-based word embeddings. In Proceedings ofthe 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2:Short Papers), volume 2, pages 302–308, 2014.[7] Jari Björne. Biomedical Event Extraction with Machine Learning. PhD thesis, TUCSDissertations, 2014.[8] Kai Hakala, Suwisa Kaewphan, Tapio Salakoski, and Filip Ginter. Syntactic analysesand named entity recognition for PubMed and PubMed Central –up-to-the-minute. InProceedings of the 2016 Workshop on Biomedical Natural Language Processing, pages102–107. Association for Computational Linguistics, 2016.ESANN 2020 proceedings, European Symposium on Artificial Neural Networks, Computational  Intelligence and Machine Learning.  Online event, 2-4 October 2020, i6doc.com publ., ISBN 978-2-87587-074-2. Available from http://www.i6doc.com/en/.618

ESANN 2020 - Proceedings, 28th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning

https://www.utupub.fi/bitstream/10024/161150/1/ES2020-56.pdf

ESANN 2020 - Proceedings, 28th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning

Abstract

Similar works

Full text

Available Versions

UTUPub