This thesis proposes a novel Knowledge Graph (KG) embedding model for Link Prediction (LP) for Knowledge Graph Completion (KGC). The missing links in a KG are predicted based on the
						existing contextual information as well as textual entity descriptions.
						The model outperforms the state-of-the-art (SOTA) model DKRL for
					FB15k and FB15k-237 datasets

Biswas, Russa

English

KITopen

Proceedings of the 1st Doctoral Consortium at the European Conference on Artificial Intelligence (DC-ECAI 2020), pages 69–70,Santiago de Compostela, Spain, August 29-30, 2020. Copyright held by the owner/author(s).Embedding based Link Prediction for Knowledge GraphCompletionRussa Biswas1Abstract. This thesis proposes a novel Knowledge Graph (KG) em-bedding model for Link Prediction (LP) for Knowledge Graph Com-pletion (KGC). The missing links in a KG are predicted based on theexisting contextual information as well as textual entity descriptions.The model outperforms the state-of-the-art (SOTA) model DKRL forFB15k and FB15k-237 datasets.1 INTRODUCTIONKGs are large networks of real world entities and relationships be-tween them. The facts are represented as a triple < h, r, t >, whereh and t are the head and tail entities respectively and r represents therelation between them. Despite the huge amounts of relational data,KGs are sparse and often incomplete as the links between the entitiesare missing. Furthermore, different KGs have information about thesame real world entities but the fact that these entities in differentKGs are same is missing.LP is a fundamental task of KGC that aims to estimate the likeli-hood of the existence of links between entities based on the currentobserved structure of the KG. LP task can be performed across dif-ferent KGs to predict the missing links between two same entitiesacross KGs and is also known as Entity Alignment. This thesis fo-cuses on the KGC task based on predicting the missing links withinthe KG as well as across multiple KGs.Due to the high complexity of the graph mining algorithms, the la-tent representation of a KG is learned into a low dimensional space.To-date many algorithms are proposed to learn the embeddings ofthe entities and relations into the same vector space. However, noneof the SOTA models consider the contextual information of the KGsalong with the textual entity descriptions to learn the latent represen-tation for the task of LP within the KG. This thesis focuses on propos-ing a model which takes the above described features into accountand performs the task of LP i.e., head, tail prediction as well as tripleclassification. On the other hand, due to the structural differencesamongst multiple KGs, their embedding spaces also exhibit differentcharacteristics. Therefore, for the entity alignment task, these differ-ent vector spaces generated for different KGs are to be aligned to asingle space to predict the missing links between the same entitiesacross different KGs.2 STATE OF THE ARTLink Prediction. So far, different KG embedding techniques havebeen proposed which can be categorized as translation based mod-els, semantic matching models, models incorporating entity types,1 FIZ Karlsruhe – Leibniz Institute for Information Infrastructure, KarlsruheInstitute of Technology, Institute AIFB, Germany email: russa.biswas@fiz-karlsruhe.demodels incorporating relation paths, models using logical rules, mod-els with temporal information, models using graph structures, andmodels incorporating information represented in literals and a de-tailed description of them are provided in [4], and [7]. Amonsgstthem, he translational model [2] use scoring function based on dis-tance and the translation is carried out with the help of a relation.GAKE [6] considers the contextual information by generating pathsstarting from an entity. On the other hand, DKRL [11] incorporatestextual entity descriptions in the embedding model and uses TransEas the base model.The textual entity descriptions present in the KGs provide infor-mation about the entity which might not be available otherwise in theKG. Also, the paths originating from an entity provide the structuralcontextual information about the neighboring entities. Therefore, inthis thesis, paths and entity descriptions are modeled together to learnthe embeddings of entities and relations for LP.Entity Alignment. Entity Alignment is the task of aligning the sameentities across different KGs. To do so, several embedding basedmethods have been proposed, in which a unified embedding space islearned using a set of already aligned entities and triples. A detaileddescription of these models for entity alignment is provided in [1].The challenges of these models are: (i) They are supervised and re-quire a set of aligned entities or triples as seeds for training. (ii) Someof the models require all the relations to be aligned between the KGs.However, in case of heterogeneous KGs which consist of differentsets of relations, it is a challenging task to have a pre-aligned set ofrelations. (iii) The methods lack proper mechanisms to handle multi-valued relations. This thesis proposes an entity alignment model forheterogeneous KGs with multi-valued relations based on the unsu-pervised approach, i.e. without pre-aligned seeds for training.3 RESEARCH QUESTIONS ANDCONTRIBUTIONSThis section discusses the research questions and the correspondingcontributions to address the challenges.• RQ1: Given an entity and a relation pair, how to predict the miss-ing entity in a triple?– The head or tail entity in a triple < h, r, ? > or <?, r, t > ispredicted by defining a mapping function ψ : E×R×E → R,where E and R are the set of entities and relations in the KG.A score is assigned to each triple where the higher the score ofthe triple indicates the more likely to be true.• RQ2: How to identify whether a given triple is valid or not?– This is a triple classification task, in which a binary classifier istrained to identify whether a given triple is false (0) or true (1).69Proceedings of the 1st Doctoral Consortium at the European Conference on Artificial Intelligence (DC-ECAI 2020), pages 69–70,Santiago de Compostela, Spain, August 29-30, 2020. Copyright held by the owner/author(s).• RQ3: How to predict the type information for an entity in a KG?– Entity typing or Entity Classification is the process of assigninga type to an entity. To do so, different structural and literal infor-mation have been exploited to train a multi-label classificationmodel for fine-grained entity typing.• RQ4: How to align the different embedding spaces of the KGs intoa unified vector space to identify the owl:sameAs links?– To align two different KG embedding spacesX and Y , a trans-lation function τ coupled with a rotation function θ is intro-duced. The owl:sameAs links are then to be determined byvector similarity.Therefore, the main contributions of this thesis are:• A novel KG embedding model exploiting the structural as well asthe textual entity descriptions in the KGs for head and tail predic-tion as well as triple classification.• A neural network based multi-label hierarchical classificationmodel for fine-grained entity typing using different features in theKG such as text and images along with the structural information.• A novel translational model to align the different KG embeddingspaces to identify the owl:sameAs links across multiple KGs.4 LINK PREDICTIONTo encapsulate the contextual information, random walks of 4 hopsare generated starting from each entity in the KG. Predicate Fre-quency Inverse Triple Frequency (PF-ITF) [8] is used to iden-tify the important relations for each entity. A sequence-to-sequence(seq2seq) learning based encoder-decoder model [10] is adapted tolearn the representation of the path vectors in the KGs as shown inFigure 1. Given a path sequence, which is a combination of enti-ties and the relations between them, such as {e1, r1, e2, r2, ..., en},the input to the encoder is the corresponding embeddings (computedusing TransE). These embeddings are passed through an attentionbased Bi-directional GRU which encapsulates the information for allinput elements and compresses them into a context vector which isthen passed through the decoder. A scaled dot product is employedas the attention mechanism. The representation of the textual en-tity descriptions is obtained using SBERT [9], followed by the sameencoder-decoder model. ConvE [5] is used as a base model for scor-ing. For triple classification, the vectors are passed through a CNNmodel. Both triple classification and head/tail prediction are evalu-ated for FB15k and FB15k-237 datasets and the model outperformsthe SOTA model DKRL as depicted in Table1. For the entity typingtask, a multi-label CNN model is to be built.5 FUTURE WORKEntity Alignment. This task is yet to be addressed in this thesis.However, the basic idea is to adapt MUSE [3] which is an unsu-pervised multi-lingual word embedding alignment model to the KGalignment. A translation function coupled with a rotational functionis to be used to align the related entities from different KGs. Thesame or related entities in different KGs will have overlapping infor-mation which could be exploited in an unsupervised manner.ACKNOWLEDGEMENTSThis thesis is supervised by Prof. Dr. Harald Sack and Dr. MehwishAlam.Figure 1. Encoder-Decoder ArchitectureTable 1. Results on LP with FB15k and FB15k-237 datasets.FB15kModels MR MRR Hits@1 Hits@3 Hits@10DKRL 85.5 0.311 0.192 0.359 0.548Our model(w/o Attn.)87 0.316 0.222 0.365 0.5615Our model(w Attn.)85 0.335 0.243 0.383 0.59FB15k-237DKRL 90.5 0.298 0.187 0.337 0.523Our model(w/o Attn.)90.5 0.314 0.217 0.349 0.527Our model(w Attn.)90 0.316 0.229 0.356 0.545REFERENCES[1] Russa Biswas, Mehwish Alam, and Harald Sack, ‘Is aligning embed-ding spaces a challenging task? an analysis of the existing methods’,arXiv preprint arXiv:2002.09247, (2020).[2] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston,and Oksana Yakhnenko, ‘Translating Embeddings for Modeling Multi-Relational Data’, in NIPS, (2013).[3] Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, LudovicDenoyer, and Hervé Jégou, ‘Word translation without parallel data’,arXiv preprint arXiv:1710.04087, (2017).[4] Yuanfei Dai, Shiping Wang, Neal N Xiong, and Wenzhong Guo, ‘Asurvey on knowledge graph embedding: Approaches, applications andbenchmarks’, Electronics, (2020).[5] Tim Dettmers, Minervini Pasquale, Stenetorp Pontus, and SebastianRiedel, ‘Convolutional 2d knowledge graph embeddings (2018)’, inAAAI.[6] Jun Feng, Minlie Huang, Yang Yang, and Xiaoyan Zhu, ‘GAKE: Graphaware knowledge embedding’, in COLING, (2016).[7] Genet Asefa Gesese, Russa Biswas, Mehwish Alam, and Harald Sack,‘A survey on knowledge graph embeddings with literals: Which modellinks better literal-ly?’, arXiv preprint arXiv:1910.12507, (2019).[8] Giuseppe Pirrò, ‘Explaining and suggesting relatedness in knowledgegraphs’, in ISWC 2015.[9] Nils Reimers and Iryna Gurevych, ‘Sentence-bert: Sentence embed-dings using siamese bert-networks’, in EMNLP-IJCNLP.[10] Ilya Sutskever, Oriol Vinyals, and Quoc V Le, ‘Sequence to sequencelearning with neural networks’, in NIPS, (2014).[11] Ruobing Xie, Zhiyuan Liu, Jia Jia, Huanbo Luan, and Maosong Sun,‘Representation Learning of Knowledge Graphs with Entity Descrip-tions’, in AAAI, (2016).70

Embedding based Link Prediction for Knowledge Graph Completion

Knowledge Graphs (KGs) have recently gained attention for representing knowledge about a particular domain. Since its advent, the Linked Open Data (LOD) cloud has constantly been growing containing many KGs about many different domains such as government, scholarly data, biomedical domain, etc. Apart from facilitating the inter-connectivity of datasets in the LOD cloud, KGs have been used in a variety of machine learning and Natural Language Processing (NLP) based applications. However, the information present in the KGs are sparse and are often incomplete. Predicting the missing links between the entities is necessary to overcome this issue. Moreover, in the LOD cloud, information about the same entities is available in multiple KGs in different forms. But the information that these entities are the same across KGs is missing. The main focus of this thesis is to do Knowledge Graph Completion by tackling the link prediction tasks within a KG as well as across different KGs. To do so, the latent representation of KGs in a low dimensional vector space has been exploited to predict the missing information in order to complete the KGs

https://publikationen.bibliothek.kit.edu/1000134456/119830726

Embedding based Link Prediction for Knowledge Graph Completion

Abstract

Similar works

Full text

Available Versions

KITopen

KITopen