De Baets, Bernard

Saeys, Yvan

Van de Peer, Yves

Van Landeghem, Sofie

English

Ghent University Academic Bibliography

Benchmarking machine learning techniques for the extractionof protein-protein interactions from textSofie Van Landeghem sofie.vanlandeghem@psb.ugent.beYvan Saeys yvan.saeys@psb.ugent.beYves Van de Peer yves.vandepeer@psb.ugent.beDepartment of Plant Systems Biology, VIB, 9052 Gent, BelgiumDepartment of Molecular Genetics, University of Ghent, 9052 Gent, BelgiumBernard De Baets Bernard.DeBaets@ugent.beDepartment of Applied Mathematics, Biometrics and Process Control, University of Ghent, 9000 Gent, BelgiumAbstractAccurately extracting information from textis a challenging discipline because of the com-plexity of natural language. We have studiedstate-of-the art systems that extract biolog-ical relations from research articles. It hasbecome clear that this field is still strugglingwith a heterogeneous collection of data sets,data formats and evaluation methods. Whilerecent developments look promising, there isstill plenty of room for improvement.1. IntroductionIn the field of life sciences it is vital to automaticallylink experimental results to data already published inonline literature resources. Fully automated systemsthat extract biological knowledge from text have thusbecome a necessity. We have studied the feasibility ofapplying machine learning approaches for the extrac-tion of protein-protein interactions (PPIs). During ourcomparative study, it became clear that there is a greatneed for the standardization of evaluation procedures.2. CorporaOver the past few years, different methods have beenproposed to extract biological relations from text. Thedevelopment of standard benchmarking data sets is astep forward towards meaningful comparison betweenthese systems. Such corpora include LLL, AImed andBioInfer, which have all been published in differentdataformats. Only recently, software has been intro-duced to convert these and two smaller data sets intoa common dataformat (Pyysalo et al., 2008), whichfacilitates comparison between different methods.Figure 1. Dependency parse for ‘The results show thatmyogenin heterodimerizes with E12 and E47 in vivo.’3. PPI extractionSentences selected from biomedical text usually con-tain complex structures with multiple subordinateclauses. Interacting proteins often occur in a sen-tence with some distance between them. Therefore,pattern-based approaches and algorithms using wordorder suffer from low recall. On the other hand, tech-niques solely based on co-occurrence of named entitiesexhibit low precision. To better capture the semanticsof a sentence, recent systems make use of informationderived from dependency trees (see Fig 1).By extracting properties from dependency trees, ex-plicit features can be obtained for each pair of pro-teins. These feature vectors are used by classifierssuch as decision trees, BayesNet and SVM to identifysentences which express a protein-protein interaction.Useful features relate to lexical and syntactic informa-tion about the children and ancestors of the proteinsin the tree, the presence of common interaction wordsand depth of the named entities in the tree.4. Ideas to improve benchmarking4.1. Common set of benchmark dataA comparative study between different PPI extractionsystems is a non trivial task as different studies oftenbenchmark on different data sets. The RelEx systemof Fundel et al. (2006) has been reimplemented withthe goal of evaluating it on different corpora (Pyysaloet al., 2008). An F score of 0.77 was obtained whenbenchmarking on LLL, and a score between 0.41 and0.44 when evaluated on AImed and Bioinfer. We ob-tain similar results when applying the walk kernel ofKim et al. (2008) to the AImed data set, which resultsin an F score of 0.44. In contrast, the original paperreports a score of 0.77 for the evaluation on LLL. Thisshows that for the same extraction method, perfor-mance can differ up to 36% depending on the choiceof the corpus. It is therefore meaningful to evaluatenew algorithms on a collection of different data sets.4.2. Instance extractionWhen benchmarking on the same corpus, different pre-processing steps can yield different instances. Homod-imers, which are self-interacting proteins, are some-times simply discarded. A similar issue is raised byannotations which are nested. The ability of the pre-processing techniques to deal with such annotationsinfluences the final number of instances in the data setand ultimately the performance of the system.Most corpora do not deal with the construction of neg-ative training data. It has become common practiceto adapt the closed world assumption, stating that nointeraction exists between two entities when there isno annotated evidence. Even though AImed providesan explicit set of abstracts with no annotated interac-tions, these are not always used, resulting in differentnumbers of negative instances in the training set.Ideally, abstracts for the testing phase should be com-pletely hidden during training. Saetre et al. (2008)pointed out that some evaluations suffer from an arti-ficial boost of performance by using features from thesame sentence in both training and testing steps ofthe machine learning algorithm. This boost of perfor-mance has been estimated between 10 and 20%.4.3. Counting true positivesThe definition of true positives varies between differ-ent evaluation approaches. Most approaches considerevery protein pair as an individual instance and evalu-ate whether an interaction is stated between these twoparticular entities. Some however state that an inter-action between two proteins may be expressed in thesame corpus by more than one instance. To extract atrue interaction, retrieving one such instance suffices.The latter evaluation technique exhibits higher recall.Even though this technique may be useful for the eval-uation of complete information retrieval systems, wefeel the first is more representative for the subtask ofextracting interactions between named entities fromindividual sentences.4.4. Directed interactionsFinally, the definition of PPI extraction task is notunambiguously defined across corpora. The LLL dataset and Bioinfer both consider the role of the differentproteins in their interaction and discriminate betweeneffectors and effectees. In AImed however, protein-protein interactions are considered to be symmetrical.This has led to the common practice of treating LLLannotations as symmetrical as well, resulting in artifi-cially higher precision rates.5. ConclusionsThe comparison of different PPI extraction methodsis hindered by the lack of standard evaluation proce-dures. We have pointed out the main problems forsuch a comparative study and indicated some practi-cal guidelines for setting up a meaningful evaluation.AcknowledgmentsSVL would like to thank the Special Research Fund(BOF) for funding her research. YS would like tothank the Research Foundation Flanders (FWO) forfunding his research.ReferencesFundel, K., Ku¨ffner, R., & Zimmer, R. (2006). Relex–relation extraction using dependency parse trees.Bioinformatics, 23, 365–371.Kim, S., Yoon, J., & Yang, J. (2008). Kernel ap-proaches for genic interaction extraction. Bioinfor-matics, 24, 118–126.Pyysalo, S., Airola, A., Heimonen, J., Bjrne, J., Gin-ter, F., & Salakoski, T. (2008). Comparative analy-sis of five protein-protein interaction corpora. BMCBioinformatics, 9.Saetre, R., Sagae, K., & Tsujii, J. (2008). Syntacticfeatures for protein-protein interaction extraction.Proceedings of the Second International Symposiumon Languages in Biology and Medicine (LBM2007).

Benchmarking machine learning techniques for the extraction of protein-protein interactions from text

https://biblio.ugent.be/publication/538704/file/592368.pdf

Benchmarking machine learning techniques for the extraction of protein-protein interactions from text

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography