Interpreting Embedding Models of Knowledge Bases: A Pedagogical Approach

Knowledge bases are employed in a variety of applications from natural language processing to semantic web search; alas, in practice their usefulness is hurt by their incompleteness. Embedding models attain state-of-the-art accuracy in knowledge base completion, but their predictions are notoriously hard to interpret. In this paper, we adapt "pedagogical approaches" (from the literature on neural networks) so as to interpret embedding models by extracting weighted Horn rules from them. We show how pedagogical approaches have to be adapted to take upon the large-scale relational aspects of knowledge bases and show experimentally their strengths and weaknesses.Comment: presented at 2018 ICML Workshop on Human Interpretability in Machine Learning (WHI 2018), Stockholm, Swede

A Fully Attention-Based Information Retriever

Recurrent neural networks are now the state-of-the-art in natural language processing because they can build rich contextual representations and process texts of arbitrary length. However, recent developments on attention mechanisms have equipped feedforward networks with similar capabilities, hence enabling faster computations due to the increase in the number of operations that can be parallelized. We explore this new type of architecture in the domain of question-answering and propose a novel approach that we call Fully Attention Based Information Retriever (FABIR). We show that FABIR achieves competitive results in the Stanford Question Answering Dataset (SQuAD) while having fewer parameters and being faster at both learning and inference than rival methods.Comment: Accepted for presentation at the International Joint Conference on Neural Networks (IJCNN) 201

On Pruning for Score-Based Bayesian Network Structure Learning

Many algorithms for score-based Bayesian net-work structure learning (BNSL), in particularexact ones, take as input a collection of po-tentially optimal parent sets for each variablein the data. Constructing such collectionsnaively is computationally intensive since thenumber of parent sets grows exponentiallywith the number of variables. Thus, pruningtechniques are not only desirable but essen-tial. While good pruning rules exist for theBayesian Information Criterion (BIC), currentresults for the Bayesian Dirichlet equivalentuniform (BDeu) score reduce the search spacevery modestly, hampering the use of the (oftenpreferred) BDeu. We derive new non-trivialtheoretical upper bounds for the BDeu scorethat considerably improve on the state-of-the-art. Since the new bounds are mathematicallyproven to be tighter than previous ones andat little extra computational cost, they are apromising addition to BNSL methods

An experimental study of prior dependence in Bayesian network structure learning

The Bayesian Dirichlet equivalent uniform (BDeu) function is a popular score to evaluate the goodness of a Bayesian network structure given complete categorical data. Despite its interesting properties, such as likelihood equivalence, it does require a prior expressed via a user-defined parameter known as Equivalent Sample Size (ESS), which significantly affects the final structure. We study conditions to obtain prior independence in BDeu-based structure learning. We show in experiments that the amount of data needed to render the learning robust to different ESS values is prohibitively large, even in big data times