1,039,370 research outputs found
KGLM: Integrating Knowledge Graph Structure in Language Models for Link Prediction
The ability of knowledge graphs to represent complex relationships at scale
has led to their adoption for various needs including knowledge representation,
question-answering, fraud detection, and recommendation systems. Knowledge
graphs are often incomplete in the information they represent, necessitating
the need for knowledge graph completion tasks, such as link and relation
prediction. Pre-trained and fine-tuned language models have shown promise in
these tasks although these models ignore the intrinsic information encoded in
the knowledge graph, namely the entity and relation types. In this work, we
propose the Knowledge Graph Language Model (KGLM) architecture, where we
introduce a new entity/relation embedding layer that learns to differentiate
distinctive entity and relation types, therefore allowing the model to learn
the structure of the knowledge graph. In this work, we show that further
pre-training the language models with this additional embedding layer using the
triples extracted from the knowledge graph, followed by the standard
fine-tuning phase sets a new state-of-the-art performance for the link
prediction task on the benchmark datasets
relation path embedding in knowledge graphs
Large-scale knowledge graphs have currently reached impressive sizes; however, they are still far from complete. In addition, most existing methods for knowledge graph completion only consider the direct links between entities, ignoring the vital impact of the semantics of relation paths. In this paper, we study the problem of how to better embed entities and relations of knowledge graphs into different low-dimensional spaces by taking full advantage of the additional semantics of relation paths and propose a novel relation path embedding model named as RPE. Specifically, with the corresponding relation and path projections, RPE can simultaneously embed each entity into two types of latent spaces. Moreover, type constraints are extended from traditional relation-specific type constraints to the proposed path-specific type constraints and both of the two type constraints can be seamlessly incorporated into RPE. The proposed model is evaluated on the benchmark tasks of link prediction and triple classification. The results of experiments demonstrate our method outperforms all baselines on both tasks. They indicate that our model is capable of catching the semantics of relation paths, which is significant for knowledge representation learning
End-to-end Structure-Aware Convolutional Networks for Knowledge Base Completion
Knowledge graph embedding has been an active research topic for knowledge
base completion, with progressive improvement from the initial TransE, TransH,
DistMult et al to the current state-of-the-art ConvE. ConvE uses 2D convolution
over embeddings and multiple layers of nonlinear features to model knowledge
graphs. The model can be efficiently trained and scalable to large knowledge
graphs. However, there is no structure enforcement in the embedding space of
ConvE. The recent graph convolutional network (GCN) provides another way of
learning graph node embedding by successfully utilizing graph connectivity
structure. In this work, we propose a novel end-to-end Structure-Aware
Convolutional Network (SACN) that takes the benefit of GCN and ConvE together.
SACN consists of an encoder of a weighted graph convolutional network (WGCN),
and a decoder of a convolutional network called Conv-TransE. WGCN utilizes
knowledge graph node structure, node attributes and edge relation types. It has
learnable weights that adapt the amount of information from neighbors used in
local aggregation, leading to more accurate embeddings of graph nodes. Node
attributes in the graph are represented as additional nodes in the WGCN. The
decoder Conv-TransE enables the state-of-the-art ConvE to be translational
between entities and relations while keeps the same link prediction performance
as ConvE. We demonstrate the effectiveness of the proposed SACN on standard
FB15k-237 and WN18RR datasets, and it gives about 10% relative improvement over
the state-of-the-art ConvE in terms of HITS@1, HITS@3 and [email protected]: The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI
2019
Hierarchical Losses and New Resources for Fine-grained Entity Typing and Linking
Extraction from raw text to a knowledge base of entities and fine-grained
types is often cast as prediction into a flat set of entity and type labels,
neglecting the rich hierarchies over types and entities contained in curated
ontologies. Previous attempts to incorporate hierarchical structure have
yielded little benefit and are restricted to shallow ontologies. This paper
presents new methods using real and complex bilinear mappings for integrating
hierarchical information, yielding substantial improvement over flat
predictions in entity linking and fine-grained entity typing, and achieving new
state-of-the-art results for end-to-end models on the benchmark FIGER dataset.
We also present two new human-annotated datasets containing wide and deep
hierarchies which we will release to the community to encourage further
research in this direction: MedMentions, a collection of PubMed abstracts in
which 246k mentions have been mapped to the massive UMLS ontology; and TypeNet,
which aligns Freebase types with the WordNet hierarchy to obtain nearly 2k
entity types. In experiments on all three datasets we show substantial gains
from hierarchy-aware training.Comment: ACL 201
Type-Constrained Representation Learning in Knowledge Graphs
Large knowledge graphs increasingly add value to various applications that
require machines to recognize and understand queries and their semantics, as in
search or question answering systems. Latent variable models have increasingly
gained attention for the statistical modeling of knowledge graphs, showing
promising results in tasks related to knowledge graph completion and cleaning.
Besides storing facts about the world, schema-based knowledge graphs are backed
by rich semantic descriptions of entities and relation-types that allow
machines to understand the notion of things and their semantic relationships.
In this work, we study how type-constraints can generally support the
statistical modeling with latent variable models. More precisely, we integrated
prior knowledge in form of type-constraints in various state of the art latent
variable approaches. Our experimental results show that prior knowledge on
relation-types significantly improves these models up to 77% in link-prediction
tasks. The achieved improvements are especially prominent when a low model
complexity is enforced, a crucial requirement when these models are applied to
very large datasets. Unfortunately, type-constraints are neither always
available nor always complete e.g., they can become fuzzy when entities lack
proper typing. We show that in these cases, it can be beneficial to apply a
local closed-world assumption that approximates the semantics of relation-types
based on observations made in the data
Neural Variational Inference For Estimating Uncertainty in Knowledge Graph Embeddings
Recent advances in Neural Variational Inference allowed for a renaissance in
latent variable models in a variety of domains involving high-dimensional data.
While traditional variational methods derive an analytical approximation for
the intractable distribution over the latent variables, here we construct an
inference network conditioned on the symbolic representation of entities and
relation types in the Knowledge Graph, to provide the variational
distributions. The new framework results in a highly-scalable method. Under a
Bernoulli sampling framework, we provide an alternative justification for
commonly used techniques in large-scale stochastic variational inference, which
drastically reduce training time at a cost of an additional approximation to
the variational lower bound. We introduce two models from this highly scalable
probabilistic framework, namely the Latent Information and Latent Fact models,
for reasoning over knowledge graph-based representations. Our Latent
Information and Latent Fact models improve upon baseline performance under
certain conditions. We use the learnt embedding variance to estimate predictive
uncertainty during link prediction, and discuss the quality of these learnt
uncertainty estimates. Our source code and datasets are publicly available
online at
https://github.com/alexanderimanicowenrivers/Neural-Variational-Knowledge-Graphs.Comment: Accepted at IJCAI 19 Neural-Symbolic Learning and Reasoning Worksho
Interaction Embeddings for Prediction and Explanation in Knowledge Graphs
Knowledge graph embedding aims to learn distributed representations for
entities and relations, and is proven to be effective in many applications.
Crossover interactions --- bi-directional effects between entities and
relations --- help select related information when predicting a new triple, but
haven't been formally discussed before. In this paper, we propose CrossE, a
novel knowledge graph embedding which explicitly simulates crossover
interactions. It not only learns one general embedding for each entity and
relation as most previous methods do, but also generates multiple triple
specific embeddings for both of them, named interaction embeddings. We evaluate
embeddings on typical link prediction tasks and find that CrossE achieves
state-of-the-art results on complex and more challenging datasets. Furthermore,
we evaluate embeddings from a new perspective --- giving explanations for
predicted triples, which is important for real applications. In this work, an
explanation for a triple is regarded as a reliable closed-path between the head
and the tail entity. Compared to other baselines, we show experimentally that
CrossE, benefiting from interaction embeddings, is more capable of generating
reliable explanations to support its predictions.Comment: This paper is accepted by WSDM201
- …