In this paper, we present a preliminary study to compute embeddings for OWL 2 ontologies by projecting the ontology axioms into a graph and performing (random) walks over the ontology graph to create a corpus of sentences. This corpus is then given to a neural language model to create concept embeddings. The conducted preliminary evaluation shows promising results

Chen, J.

Holter, O. M.

Jimenez-Ruiz, E.

Myklebust, E. B.

In this paper, we present a preliminary study to compute embeddings for OWL 2 ontologies by projecting the ontology axioms into a graph and performing (random) walks over the ontology graph to create a corpus of sentences. This corpus is then given to a neural language model to create concept embeddings. The conducted preliminary evaluation shows promising results.publishedVersio

Holter, Ole Magnus

Myklebust, Erik Bryhn

Chen, Jiaoyan

Jimenez-Ruiz, Ernesto

NIVA Open Access Archive

Embedding OWL Ontologies with OWL2Vec?Ole Magnus Holter1, Erik B. Myklebust1,2, Jiaoyan Chen3, andErnesto Jimenez-Ruiz??1,41 Department of Informatics, University of Oslo, Norway2 Norwegian Institute for Water Research, Oslo, Norway3 Department of Computer Science, University of Oxford, UK4 The Alan Turing Institute, London, UKAbstract. In this paper, we present a preliminary study to compute embeddingsfor OWL 2 ontologies by projecting the ontology axioms into a graph and per-forming (random) walks over the ontology graph to create a corpus of sentences.This corpus is then given to a neural language model to create concept embed-dings. The conducted preliminary evaluation shows promising results.1 IntroductionIn the literature we can find a number of approaches that perform embeddings over(RDF) knowledge graphs [17] to conduct knowledge graph completion (e.g., [4, 11,14]). Most of the approaches, however, only focus on the embedding of the data in-stances. Although some approaches also learn embeddings for concepts involved ininstance type definitions (e.g., [11, 8]), the embeddings rely on data instances and theknowledge provided by the ontology (e.g., subsumption axioms) is typically ignored.Alshahrani et al. [2] performs reasoning to expand the knowledge graph with new facts(e.g., types of the instances), but the main focus is on the instance embeddings for bio-logical link prediction.Regarding concept embeddings, there have been some efforts to leverage word em-beddings to associate a vector to the lexical information of the ontology concepts. Thisapproach has typically been applied to ontology alignment tasks (e.g., [13, 9]). Themain limitation of this approach is the dependence on a relevant text corpus or a pre-trained set of word embeddings, which may have some limitations when applying toontologies with domain-specific vocabulary. Some works refine the word embeddingsusing semantic lexicons (e.g., [9]) to compensate for the lack of domain-specific train-ing corpora. Nevertheless, the computed word embeddings neglect the rich semanticsof the ontologies (e.g., concept hierarchy, relationships among concepts).The approach followed by the systems Onto2Vec [15] and OPA2Vec [16] deservesspecial mention. Both Onto2Vec and OPA2Vec consider each axiom in the ontology asa sentence. The set of axioms (including some inferred axioms) in the ontology forma document that is then given to Word2Vec [10]. Word2Vec computes vectors for eachof the elements in the document including concept identifiers, relationships and OWL? Copyright ©2019 for this paper by its authors. Use permitted under Creative Commons Li-cense Attribution 4.0 International (CC BY 4.0).?? Corresponding author: ernesto.jimenez.ruiz@gmail.com2 O. M. Holter et al.Fig. 1: Current pipeline in OWL2Vecconstructs. Although this approach represents an interesting effort, it has the followinglimitations: (i) the corpus of sentences may be limited for small-medium ontologies tocreate meaningful vectors, (ii) OWL constructs may introduce noise in the embeddings,and (iii) Word2Vec does not differentiate between sentences like “A SubClassOf: B”and “A DisjointWith: C” which will lead to similar embeddings for A, B and C.In order to overcome the limitations of state-of-the-art approaches, we have imple-mented a framework to compute semantic embeddings from OWL 2 ontologies. Ourapproach (i) projects the ontology into a graph, (ii) implements several strategies towalk the ontology graph, (iii) creates a corpus of sentences according to the walkingstrategies, and (iv) generates concept embeddings from that corpus.2 MethodsFigure 1 summarises the current architecture of the OWL2Vec framework, composed ofthree main components: ontology projection, walk strategy, and concept embeddings.Ontology projection. We follow a simplified version of the (RDF-based) graph pro-jection of the ontology used by Agibetov et al. [1]. The nodes in the projected RDFgraph represent concepts in the ontology while edges are labelled with possible rela-tions among those concepts. The key property of this projection is that every edge (i.e.,triple 〈A,Ro, B〉) in the graph is justified by one or more axioms entailed by the on-tology which “semantically relates” two concepts (e.g., A and B) via a property (e.g.,Ro). Table 1 shows the type of axioms currently considered in the ontology projection.Walk strategy. We have implemented a set of strategies to walk the ontology graph.We initially relied on a modified version of RDF2Vec [14]. The main difference withrespect to the original RDF2Vec algorithm is the use of the ontology projection as in-put and the inclusion of weighted edges for the walks (as also proposed in [5]). Onecould give more weight to the taxonomic relationships or to the object properties towalk from one hierarchy branch to another. The modified algorithm also allowed thecreation of sentences with the concept URI and/or the concept labels. We encountered,however, a scalability limitation for long walks over large ontologies. To overcomethe limitations of the RDF2Vec approach, we implemented a more flexible strategy in-spired by node2vec [6]. This strategy (i) scales with large ontologies, (ii) allows to biasthe walks, (iii) enables semantic similarity not only for closely connected elements butalso for similar structures, (iv) has flexibility to change the direction of a walk to avoidEmbedding OWL Ontologies with OWL2Vec 3Condition 1 Condition 2 Triple(s)A SubClassOf : Ro Restriction D D ≡ B |B1 unionsq ... unionsq Bn |B1 u ... u Bn〈A,Ro, B〉 orRo Restriction D SubClassOf : ARo Domain: A Ro Range: B 〈A,Ro, Bi〉 for i∈1..nA SubClassOf : Ro value b b type BRo InverseOf : R−o 〈A,R−o , B〉 in graphS1 ◦ ... ◦ Sn SubPropertyOf : Ro 〈A,S1, C1〉...〈Cn, Sn, B〉 in graphB SubClassOf : A〈B,SubClassOf,A〉〈A,SubClassOf−, B〉Table 1: Projection of the OWL 2 ontology axioms into an RDF graph. Restriction isone of: min, max, exactly, some, only. A, B, Bi and Ci are atomic concepts, Si, Ro andR−o are object properties, b an individual.dead ends, and (v) uses the (in-memory) triple store reasoner RDFox [12] to enhancethe access to the projected ontology graph.Concept embeddings. The walk strategies in OWL2Vec are flexible and allow the cre-ation of different types of corpora of sentences that will lead to concept embeddingswith different characteristics. For example, the computed embeddings may favour thesemantic similarity among concepts within the same hierarchy (e.g., between PersonandResearcher) or among concepts related with other properties (e.g., betweenPaperandResearcher). We currently rely on Word2Vec [10] and FastText [7] to compute theembedding from the resulting documents.3 Preliminary Evaluation and Future WorkFigure 2 shows a subset of our preliminary set of experiments.5 We have computed(agglomerative) clusters of the concepts in the EKAW conference ontology based onthe embeddings provided by RDF2Vec, Onto2Vec and OWL2Vec. We can observe thatthe clusters (of related concepts) obtained with the OWL2Vec embeddings are welldifferentiated while for RDF2Vec and Onto2Vec the cloud of points is more sparse.6These clustering results are encouraging, but more evaluation is required to evaluatethe usefulness of the computed embeddings. We plan to conduct an extensive evaluationto obtain quality measures similar to the ones proposed within the Concept2vec frame-work [3]. We also aim at evaluating OWL2Vec in real-world applications like biomedi-cal link prediction or ecotoxicological effect prediction to analyze if the OWL2Vec con-cept embeddings improve the state-of-the-art solutions. Furthermore, we are adoptingOWL2Vec within our ontology alignment system as the different OWL2Vec walkingstrategies has led to a promising set of concept similarities.OWL2Vec has the potential of becoming an essential component of machine learn-ing applications that rely on the semantic information of an ontology as input.Acknowledgements. This work is supported by the AIDA project (The Turing Institute)and the SIRIUS Centre for Scalable Data Access (RCN 237889).5 OWL2Vec source codes available from: https://gitlab.com/oholter/owl2vec6 The rest of conference-based OntoFarm ontologies [18] led to similar findings.4 O. M. Holter et al.(a) RDF2Vec (b) Onto2Vec (c) OWL2VecFig. 2: Clustering of the EKAW conference ontology.References1. Agibetov, A., et al.: Supporting shared hypothesis testing in the biomedical domain. J.Biomedical Semantics 9(1), 9:1–9:22 (2018)2. Alshahrani, M., et al.: Neuro-symbolic representation learning on biological knowledgegraphs. Bioinformatics 33(17), 2723–2730 (2017)3. Alshargi, F., Shekarpour, S., Soru, T., Sheth, A.P., Quasthoff, U.: Concept2vec: Metrics forEvaluating Quality of Embeddings for Ontological Concepts. CoRR abs/1803.04488 (2018)4. Bordes, A., et al.: Translating Embeddings for Modeling Multi-relational Data. In: 27th Con-ference on Neural Information Processing Systems (NIPS) (2013)5. Cochez, M., Ristoski, P., Ponzetto, S.P., Paulheim, H.: Global RDF vector space embeddings.In: International Semantic Web Conference (ISWC). pp. 190–207 (2017)6. Grover, A., Leskovec, J.: node2vec: Scalable Feature Learning for Networks. In: SIGKDDInternational Conference on Knowledge Discovery and Data Mining. pp. 855–864 (2016)7. Joulin, A., Grave, E., Bojanowski, P., Douze, M., Je´gou, H., Mikolov, T.: FastText.zip: Com-pressing text classification models. CoRR abs/1612.03651 (2016)8. Kejriwal, M., Szekely, P.: Scalable Generation of Type Embeddings Using the ABox. OJSW4(1), 20–34 (2017)9. Kolyvakis, P., et al.: DeepAlignment: Unsupervised Ontology Matching with Refined WordVectors. In: NAACL Conference on Human Language Technologies (2018)10. Mikolov, T., et al.: Distributed Representations of Words and Phrases and their Composition-ality. In: 27th Conference on Neural Information Processing Systems (NIPS) (2013)11. Moon, C., Jones, P., Samatova, N.F.: Learning entity type embeddings for knowledge graphcompletion. In: Conference on Information and Knowledge Management (CIKM) (2017)12. Nenov, Y., Piro, R., Motik, B., Horrocks, I., Wu, Z., Banerjee, J.: RDFox: A Highly-ScalableRDF Store. In: 14th International Semantic Web Conference (2015)13. Nkisi-Orji, I., Wiratunga, N., Massie, S., Hui, K., Heaven, R.: Ontology Alignment Based onWord Embedding and Random Forest Classification. In: ECML-PKDD. pp. 557–572 (2018)14. Ristoski, P., Rosati, J., Noia, T.D., Leone, R.D., Paulheim, H.: RDF2Vec: RDF graph em-beddings and their applications. Semantic Web 10(4), 721–752 (2019)15. Smaili, F.Z., Gao, X., Hoehndorf, R.: Onto2Vec: joint vector-based representation of biolog-ical entities and their ontology-based annotations. Bioinformatics 34(13), i52–i60 (2018)16. Smaili, F.Z., Gao, X., Hoehndorf, R.: OPA2Vec: combining formal and informal content ofbiomedical ontologies to improve similarity-based prediction. Bioinformatics 35(12) (2019)17. Wang, Q., Mao, Z., Wang, B., Guo, L.: Knowledge Graph Embedding: A Survey of Ap-proaches and Applications. IEEE Trans. Knowl. Data Eng. 29(12), 2724–2743 (2017)18. Zamazal, O., Sva´tek, V.: The Ten-Year OntoFarm and its Fertilization within the Onto-Sphere. J. Web Semant. 43, 46–53 (2017)

English

Embedding OWL ontologies with OWL2Vec

NORA - Norwegian Open Research Archives

City Research Online

              City, University of London Institutional RepositoryCitation: Holter, O. M., Myklebust, E. B., Chen, J. and Jimenez-Ruiz, E. (2019). Embedding OWL ontologies with OWL2Vec. CEUR Workshop Proceedings, 2456, pp. 33-36. This is the accepted version of the paper. This version of the publication may differ from the final published version. Permanent repository link:  http://openaccess.city.ac.uk/id/eprint/23181/Link to published version: Copyright and reuse: City Research Online aims to make research outputs of City, University of London available to a wider audience. Copyright and Moral Rights remain with the author(s) and/or copyright holders. URLs from City Research Online may be freely distributed and linked to.City Research Online:            http://openaccess.city.ac.uk/            publications@city.ac.ukCity Research OnlineEmbedding OWL ontologies with OWL2Vec?Ole Magnus Holter1, Erik B. Myklebust1,2, Jiaoyan Chen3, andErnesto Jimenez-Ruiz??1,41 Department of Informatics, University of Oslo, Norway2 Norwegian Institute for Water Research, Oslo, Norway3 Department of Computer Science, University of Oxford, UK4 The Alan Turing Institute, London, UKAbstract. In this paper, we present a preliminary study to compute embeddingsfor OWL 2 ontologies by projecting the ontology axioms into a graph and per-forming (random) walks over the ontology graph to create a corpus of sentences.This corpus is then given to a neural language model to create concept embed-dings. The conducted preliminary evaluation shows promising results.1 IntroductionIn the literature we can find a number of approaches that perform embeddings over(RDF) knowledge graphs [17] to conduct knowledge graph completion (e.g., [4, 11,14]). Most of the approaches, however, only focus on the embedding of the data in-stances. Although some approaches also learn embeddings for concepts involved ininstance type definitions (e.g., [11, 8]), the embeddings rely on data instances and theknowledge provided by the ontology (e.g., subsumption axioms) is typically ignored.Alshahrani et al. [2] performs reasoning to expand the knowledge graph with new facts(e.g., types of the instances), but the main focus is on the instance embeddings for bio-logical link prediction.Regarding concept embeddings, there have been some efforts to leverage word em-beddings to associate a vector to the lexical information of the ontology concepts. Thisapproach has typically been applied to ontology alignment tasks (e.g., [13, 9]). Themain limitation of this approach is the dependence on a relevant text corpus or a pre-trained set of word embeddings, which may have some limitations when applying toontologies with domain-specific vocabulary. Some works refine the word embeddingsusing semantic lexicons (e.g., [9]) to compensate for the lack of domain-specific train-ing corpora. Nevertheless, the computed word embeddings neglect the rich semanticsof the ontologies (e.g., concept hierarchy, relationships among concepts).The approach followed by the systems Onto2Vec [15] and OPA2Vec [16] deservesspecial mention. Both Onto2Vec and OPA2Vec consider each axiom in the ontology asa sentence. The set of axioms (including some inferred axioms) in the ontology forma document that is then given to Word2Vec [10]. Word2Vec computes vectors for eachof the elements in the document including concept identifiers, relationships and OWL? Copyright ©2019 for this paper by its authors. Use permitted under Creative Commons Li-cense Attribution 4.0 International (CC BY 4.0).?? Corresponding author: ernesto.jimenez.ruiz@gmail.com2 O. M. Holter et al.Fig. 1: Current pipeline in OWL2Vecconstructs. Although this approach represents an interesting effort, it has the followinglimitations: (i) the corpus of sentences may be limited for small-medium ontologies tocreate meaningful vectors, (ii) OWL constructs may introduce noise in the embeddings,and (iii) Word2Vec does not differentiate between sentences like “A SubClassOf: B”and “A DisjointWith: C” which will lead to similar embeddings for A, B and C.In order to overcome the limitations of state-of-the-art approaches, we have imple-mented a framework to compute semantic embeddings from OWL 2 ontologies. Ourapproach (i) projects the ontology into a graph, (ii) implements several strategies towalk the ontology graph, (iii) creates a corpus of sentences according to the walkingstrategies, and (iv) generates concept embeddings from that corpus.2 MethodsFigure 1 summarises the current architecture of the OWL2Vec framework, composed ofthree main components: ontology projection, walk strategy, and concept embeddings.Ontology projection. We follow a simplified version of the (RDF-based) graph pro-jection of the ontology used by Agibetov et al. [1]. The nodes in the projected RDFgraph represent concepts in the ontology while edges are labelled with possible rela-tions among those concepts. The key property of this projection is that every edge (i.e.,triple 〈A,Ro, B〉) in the graph is justified by one or more axioms entailed by the on-tology which “semantically relates” two concepts (e.g., A and B) via a property (e.g.,Ro). Table 1 shows the type of axioms currently considered in the ontology projection.Walk strategy. We have implemented a set of strategies to walk the ontology graph.We initially relied on a modified version of RDF2Vec [14]. The main difference withrespect to the original RDF2Vec algorithm is the use of the ontology projection as in-put and the inclusion of weighted edges for the walks (as also proposed in [5]). Onecould give more weight to the taxonomic relationships or to the object properties towalk from one hierarchy branch to another. The modified algorithm also allowed thecreation of sentences with the concept URI and/or the concept labels. We encountered,however, a scalability limitation for long walks over large ontologies. To overcomethe limitations of the RDF2Vec approach, we implemented a more flexible strategy in-spired by node2vec [6]. This strategy (i) scales with large ontologies, (ii) allows to biasthe walks, (iii) enables semantic similarity not only for closely connected elements butalso for similar structures, (iv) has flexibility to change the direction of a walk to avoidEmbedding OWL ontologies with OWL2Vec 3Condition 1 Condition 2 Triple(s)A SubClassOf : Ro Restriction D D ≡ B |B1 unionsq ... unionsq Bn |B1 u ... u Bn〈A,Ro, B〉 orRo Restriction D SubClassOf : ARo Domain: A Ro Range: B 〈A,Ro, Bi〉 for i∈1..nA SubClassOf : Ro value b b type BRo InverseOf : R−o 〈A,R−o , B〉 in graphS1 ◦ ... ◦ Sn SubPropertyOf : Ro 〈A,S1, C1〉...〈Cn, Sn, B〉 in graphB SubClassOf : A〈B,SubClassOf,A〉〈A,SubClassOf−, B〉Table 1: Projection of the OWL 2 ontology axioms into an RDF graph. Restriction isone of: min, max, exactly, some, only. A, B, Bi and Ci are atomic concepts, Si, Ro andR−o are object properties, b an individual.dead ends, and (v) uses the (in-memory) triple store reasoner RDFox [12] to enhancethe access to the projected ontology graph.Concept embeddings. The walk strategies in OWL2Vec are flexible and allow the cre-ation of different types of corpora of sentences that will lead to concept embeddingswith different characteristics. For example, the computed embeddings may favour thesemantic similarity among concepts within the same hierarchy (e.g., between PersonandResearcher) or among concepts related with other properties (e.g., betweenPaperandResearcher). We currently rely on Word2Vec [10] and FastText [7] to compute theembedding from the resulting documents.3 Preliminary Evaluation and Future WorkFigure 2 shows a subset of our preliminary set of experiments.5 We have computed(agglomerative) clusters of the concepts in the EKAW conference ontology based onthe embeddings provided by RDF2Vec, Onto2Vec and OWL2Vec. We can observe thatthe clusters (of related concepts) obtained with the OWL2Vec embeddings are welldifferentiated while for RDF2Vec and Onto2Vec the cloud of points is more sparse.6These clustering results are encouraging, but more evaluation is required to evaluatethe usefulness of the computed embeddings. We plan to conduct an extensive evaluationto obtain quality measures similar to the ones proposed within the Concept2vec frame-work [3]. We also aim at evaluating OWL2Vec in real-world applications like biomedi-cal link prediction or ecotoxicological effect prediction to analyze if the OWL2Vec con-cept embeddings improve the state-of-the-art solutions. Furthermore, we are adoptingOWL2Vec within our ontology alignment system as the different OWL2Vec walkingstrategies has led to a promising set of concept similarities.OWL2Vec has the potential of becoming an essential component of machine learn-ing applications that rely on the semantic information of an ontology as input.Acknowledgements. This work is supported by the AIDA project (The Turing Institute)and the SIRIUS Centre for Scalable Data Access (RCN 237889).5 OWL2Vec source codes available from: https://gitlab.com/oholter/owl2vec6 The rest of conference-based OntoFarm ontologies [18] led to similar findings.4 O. M. Holter et al.(a) RDF2Vec (b) Onto2Vec (c) OWL2VecFig. 2: Clustering of the EKAW conference ontology.References1. Agibetov, A., et al.: Supporting shared hypothesis testing in the biomedical domain. J.Biomedical Semantics 9(1), 9:1–9:22 (2018)2. Alshahrani, M., et al.: Neuro-symbolic representation learning on biological knowledgegraphs. Bioinformatics 33(17), 2723–2730 (2017)3. Alshargi, F., Shekarpour, S., Soru, T., Sheth, A.P., Quasthoff, U.: Concept2vec: Metrics forEvaluating Quality of Embeddings for Ontological Concepts. CoRR abs/1803.04488 (2018)4. Bordes, A., et al.: Translating Embeddings for Modeling Multi-relational Data. In: 27th Con-ference on Neural Information Processing Systems (NIPS) (2013)5. Cochez, M., Ristoski, P., Ponzetto, S.P., Paulheim, H.: Global RDF vector space embeddings.In: International Semantic Web Conference (ISWC). pp. 190–207 (2017)6. Grover, A., Leskovec, J.: node2vec: Scalable Feature Learning for Networks. In: SIGKDDInternational Conference on Knowledge Discovery and Data Mining. pp. 855–864 (2016)7. Joulin, A., Grave, E., Bojanowski, P., Douze, M., Je´gou, H., Mikolov, T.: FastText.zip: Com-pressing text classification models. CoRR abs/1612.03651 (2016)8. Kejriwal, M., Szekely, P.: Scalable Generation of Type Embeddings Using the ABox. OJSW4(1), 20–34 (2017)9. Kolyvakis, P., et al.: DeepAlignment: Unsupervised Ontology Matching with Refined WordVectors. In: NAACL Conference on Human Language Technologies (2018)10. Mikolov, T., et al.: Distributed Representations of Words and Phrases and their Composition-ality. In: 27th Conference on Neural Information Processing Systems (NIPS) (2013)11. Moon, C., Jones, P., Samatova, N.F.: Learning entity type embeddings for knowledge graphcompletion. In: Conference on Information and Knowledge Management (CIKM) (2017)12. Nenov, Y., Piro, R., Motik, B., Horrocks, I., Wu, Z., Banerjee, J.: RDFox: A Highly-ScalableRDF Store. In: 14th International Semantic Web Conference (2015)13. Nkisi-Orji, I., Wiratunga, N., Massie, S., Hui, K., Heaven, R.: Ontology Alignment Based onWord Embedding and Random Forest Classification. In: ECML-PKDD. pp. 557–572 (2018)14. Ristoski, P., Rosati, J., Noia, T.D., Leone, R.D., Paulheim, H.: RDF2Vec: RDF graph em-beddings and their applications. Semantic Web 10(4), 721–752 (2019)15. Smaili, F.Z., Gao, X., Hoehndorf, R.: Onto2Vec: joint vector-based representation of biolog-ical entities and their ontology-based annotations. Bioinformatics 34(13), i52–i60 (2018)16. Smaili, F.Z., Gao, X., Hoehndorf, R.: OPA2Vec: combining formal and informal content ofbiomedical ontologies to improve similarity-based prediction. Bioinformatics 35(12) (2019)17. Wang, Q., Mao, Z., Wang, B., Guo, L.: Knowledge Graph Embedding: A Survey of Ap-proaches and Applications. IEEE Trans. Knowl. Data Eng. 29(12), 2724–2743 (2017)18. Zamazal, O., Sva´tek, V.: The Ten-Year OntoFarm and its Fertilization within the Onto-Sphere. J. Web Semant. 43, 46–53 (2017)

http://openaccess.city.ac.uk/id/eprint/23181/1/OWL2vec_iswc2019_poster.pdf

Embedding OWL ontologies with OWL2Vec

Abstract

Similar works

Full text

Available Versions

NIVA Open Access Archive

NORA - Norwegian Open Research Archives

City Research Online

NORA - Norwegian Open Research Archives