Unsupervised Learning of Relational Entailment Graphs from Text

Recognizing textual entailment and paraphrasing is critical to many core natural language processing applications including question answering and semantic parsing. The surface form of a sentence that answers a question such as “Does Facebook own Instagram?” frequently does not directly correspond to the form of the question, but is rather a paraphrase or an expression such as “Facebook bought Instagram”, that entails the answer. Relational entailments (e.g., buys entails owns) are crucial for bridging the gap between queries and text resources. In this thesis, we describe different unsupervised approaches to construct relational entailment graphs, with typed relations (e.g., company buys company) as nodes and entailment as directed edges. The entailment graphs provide an explainable resource for downstream tasks such as question answering; however, the existing methods suffer from noise and sparsity inherent to the data. We extract predicate-argument structures from large multiple-source news corpora using a fast Combinatory Categorial Grammar parser. We compute entailment scores between relations based on the Distributional Inclusion Hypothesis which states that a word (relation) p entails another word (relation) q if and only if in any context that p can be used, q can be used in its place. The entailment scores are used to build local entailment graphs. We then build global entailment graphs by exploiting the dependencies between the entailment rules. Previous work has used transitivity constraints, but these constraints are intractable on large graphs. We instead propose a scalable method that learns globally consistent similarity scores based on new soft constraints that consider both the structures across typed entailment graphs and inside each graph. We show that our method significantly improves the entailment graphs. Additionally, we show the duality of entailment graph induction with the task of link prediction. The link prediction task infers missing relations between entities in an incomplete knowledge graph and discovers new facts. We present a new method in which link prediction on the knowledge graph of assertions extracted from raw text is used to improve entailment graphs which are learned from the same text. The entailment graphs are in turn used to improve the link prediction task. Finally, we define the contextual link prediction task that uses both the structure of the knowledge graph of assertions and their textual contexts. We fine-tune pre-trained language models with an unsupervised contextual link prediction objective. We augment the existing assertions with novel predictions of our model and use them to build higher quality entailment graphs. Similarly, we show that the entailment graphs improve the contextual link prediction task

Tensors and tensor decompositions for combining external information with knowledge graph embeddings

The task of knowledge graph (KG) completion, where one is given an incomplete KG as a list of facts, and is asked to give high scores to correct but unseen triples, has been a well-studied problem in the NLP community. A simple but surprisingly robust approach for solving this task emerged as learning low dimensional embeddings for entities and relations by approximating the underlying KG directly through a scoring function. Knowledge graphs have a natural representation as a binary three way array, also known as a 3rd order tensor, and certain classes of scoring functions can be characterized as finding a low-rank decomposition of this tensor. This dissertation extends this characterization, and investigates the suitability of tensors for modelling both knowledge graphs and related data, for learning low-rank representations of entities and relations that incorporate information from heterogeneous sources, and for reasoning with paths and rules using the learned representations. Specifically, we present two joint tensor decomposition models for integrating external information in the process of learning KG embeddings. Our first model is a joint tensor-tensor decomposition model that learns representations based on both KG facts and type information on entities and relations. Our second model is a joint tensor-matrix decomposition for integrating cooccurrence information between entities and words from an entity linked corpus into knowledge graph embeddings, in order to learn better representations for the entities that are rarely seen in the knowledge graph. We also investigate tensors as tools for enabling multi-step reasoning using learned embedding representations. To this end, we extend theoretical results for semiring weighted logic programs to tensors of semirings. Our results are broadly applicable to any area that uses dynamic programming algorithms for calculating tensor values. Such applications include incorporating embeddings of paths and rules for knowledge graph completion, and syntactic parsing with latent variable grammar