740 research outputs found
Knowledge-based Biomedical Data Science 2019
Knowledge-based biomedical data science (KBDS) involves the design and
implementation of computer systems that act as if they knew about biomedicine.
Such systems depend on formally represented knowledge in computer systems,
often in the form of knowledge graphs. Here we survey the progress in the last
year in systems that use formally represented knowledge to address data science
problems in both clinical and biological domains, as well as on approaches for
creating knowledge graphs. Major themes include the relationships between
knowledge graphs and machine learning, the use of natural language processing,
and the expansion of knowledge-based approaches to novel domains, such as
Chinese Traditional Medicine and biodiversity.Comment: Manuscript 43 pages with 3 tables; Supplemental material 43 pages
with 3 table
Distributed representations for multilingual language processing
Distributed representations are a central element in natural language processing. Units of text such as words, ngrams, or characters are mapped to real-valued vectors so that they can be processed by computational models. Representations trained on large amounts of text, called static word embeddings, have been found to work well across a variety of tasks such as sentiment analysis or named entity recognition. More recently, pretrained language models are used as contextualized representations that have been found to yield even better task performances.
Multilingual representations that are invariant with respect to languages are useful for multiple reasons. Models using those representations would only require training data in one language and still generalize across multiple languages. This is especially useful for languages that exhibit data sparsity. Further, machine translation models can benefit from source and target representations in the same space. Last, knowledge extraction models could not only access English data, but data in any natural language and thus exploit a richer source of knowledge.
Given that several thousand languages exist in the world, the need for multilingual language processing seems evident. However, it is not immediately clear, which properties multilingual embeddings should exhibit, how current multilingual representations work and how they could be improved.
This thesis investigates some of these questions. In the first publication, we explore the boundaries of multilingual representation learning by creating an embedding space across more than one thousand languages. We analyze existing methods and propose concept based embedding learning methods. The second paper investigates differences between creating representations for one thousand languages with little data versus considering few languages with abundant data. In the third publication, we refine a method to obtain interpretable subspaces of embeddings. This method can be used to investigate the workings of multilingual representations. The fourth publication finds that multilingual pretrained language models exhibit a high degree of multilinguality in the sense that high quality word alignments can be easily extracted. The fifth paper investigates reasons why multilingual pretrained language models are multilingual despite lacking any kind of crosslingual supervision during training. Based on our findings we propose a training scheme that leads to improved multilinguality. Last, the sixth paper investigates the use of multilingual pretrained language models as multilingual knowledge bases
Neurosymbolic AI for Reasoning on Graph Structures: A Survey
Neurosymbolic AI is an increasingly active area of research which aims to
combine symbolic reasoning methods with deep learning to generate models with
both high predictive performance and some degree of human-level
comprehensibility. As knowledge graphs are becoming a popular way to represent
heterogeneous and multi-relational data, methods for reasoning on graph
structures have attempted to follow this neurosymbolic paradigm. Traditionally,
such approaches have utilized either rule-based inference or generated
representative numerical embeddings from which patterns could be extracted.
However, several recent studies have attempted to bridge this dichotomy in ways
that facilitate interpretability, maintain performance, and integrate expert
knowledge. Within this article, we survey a breadth of methods that perform
neurosymbolic reasoning tasks on graph structures. To better compare the
various methods, we propose a novel taxonomy by which we can classify them.
Specifically, we propose three major categories: (1) logically-informed
embedding approaches, (2) embedding approaches with logical constraints, and
(3) rule-learning approaches. Alongside the taxonomy, we provide a tabular
overview of the approaches and links to their source code, if available, for
more direct comparison. Finally, we discuss the applications on which these
methods were primarily used and propose several prospective directions toward
which this new field of research could evolve.Comment: 21 pages, 8 figures, 1 table, currently under review. Corresponding
GitHub page here: https://github.com/NeSymGraph
Reasoning on Knowledge Graphs with Debate Dynamics
We propose a novel method for automatic reasoning on knowledge graphs based
on debate dynamics. The main idea is to frame the task of triple classification
as a debate game between two reinforcement learning agents which extract
arguments -- paths in the knowledge graph -- with the goal to promote the fact
being true (thesis) or the fact being false (antithesis), respectively. Based
on these arguments, a binary classifier, called the judge, decides whether the
fact is true or false. The two agents can be considered as sparse, adversarial
feature generators that present interpretable evidence for either the thesis or
the antithesis. In contrast to other black-box methods, the arguments allow
users to get an understanding of the decision of the judge. Since the focus of
this work is to create an explainable method that maintains a competitive
predictive accuracy, we benchmark our method on the triple classification and
link prediction task. Thereby, we find that our method outperforms several
baselines on the benchmark datasets FB15k-237, WN18RR, and Hetionet. We also
conduct a survey and find that the extracted arguments are informative for
users.Comment: AAAI-202
From Word to Sense Embeddings: A Survey on Vector Representations of Meaning
Over the past years, distributed semantic representations have proved to be
effective and flexible keepers of prior knowledge to be integrated into
downstream applications. This survey focuses on the representation of meaning.
We start from the theoretical background behind word vector space models and
highlight one of their major limitations: the meaning conflation deficiency,
which arises from representing a word with all its possible meanings as a
single vector. Then, we explain how this deficiency can be addressed through a
transition from the word level to the more fine-grained level of word senses
(in its broader acceptation) as a method for modelling unambiguous lexical
meaning. We present a comprehensive overview of the wide range of techniques in
the two main branches of sense representation, i.e., unsupervised and
knowledge-based. Finally, this survey covers the main evaluation procedures and
applications for this type of representation, and provides an analysis of four
of its important aspects: interpretability, sense granularity, adaptability to
different domains and compositionality.Comment: 46 pages, 8 figures. Published in Journal of Artificial Intelligence
Researc
- …