3 research outputs found
Weight Priors for Learning Identity Relations
Learning abstract and systematic relations has been an open issue in neural
network learning for over 30 years. It has been shown recently that neural
networks do not learn relations based on identity and are unable to generalize
well to unseen data. The Relation Based Pattern (RBP) approach has been
proposed as a solution for this problem. In this work, we extend RBP by
realizing it as a Bayesian prior on network weights to model the identity
relations. This weight prior leads to a modified regularization term in
otherwise standard network learning. In our experiments, we show that the
Bayesian weight priors lead to perfect generalization when learning identity
based relations and do not impede general neural network learning. We believe
that the approach of creating an inductive bias with weight priors can be
extended easily to other forms of relations and will be beneficial for many
other learning tasks.Comment: Proceedings of KR2ML @ NeurIPS 2019, Vancouver, Canad
Relational Weight Priors in Neural Networks for Abstract Pattern Learning and Language Modelling
Deep neural networks have become the dominant approach in natural language
processing (NLP). However, in recent years, it has become apparent that there
are shortcomings in systematicity that limit the performance and data
efficiency of deep learning in NLP. These shortcomings can be clearly shown in
lower-level artificial tasks, mostly on synthetic data. Abstract patterns are
the best known examples of a hard problem for neural networks in terms of
generalisation to unseen data. They are defined by relations between items,
such as equality, rather than their values. It has been argued that these
low-level problems demonstrate the inability of neural networks to learn
systematically. In this study, we propose Embedded Relation Based Patterns
(ERBP) as a novel way to create a relational inductive bias that encourages
learning equality and distance-based relations for abstract patterns. ERBP is
based on Relation Based Patterns (RBP), but modelled as a Bayesian prior on
network weights and implemented as a regularisation term in otherwise standard
network learning. ERBP is is easy to integrate into standard neural networks
and does not affect their learning capacity. In our experiments, ERBP priors
lead to almost perfect generalisation when learning abstract patterns from
synthetic noise-free sequences. ERBP also improves natural language models on
the word and character level and pitch prediction in melodies with RNN, GRU and
LSTM networks. We also find improvements in in the more complex tasks of
learning of graph edit distance and compositional sentence entailment. ERBP
consistently improves over RBP and over standard networks, showing that it
enables abstract pattern learning which contributes to performance in natural
language tasks.Comment: 29 page
A Computational Model of Infant Learning and Reasoning with Probabilities
Recent experiments reveal that 6- to 12-month-old infants can learn
probabilities and reason with them. In this work, we present a novel
computational system called Neural Probability Learner and Sampler (NPLS) that
learns and reasons with probabilities, providing a computationally sufficient
mechanism to explain infant probabilistic learning and inference. In 24
computer simulations, NPLS simulations show how probability distributions can
emerge naturally from neural-network learning of event sequences, providing a
novel explanation of infant probabilistic learning and reasoning. Three
mathematical proofs show how and why NPLS simulates the infant results so
accurately. The results are situated in relation to seven other active research
lines. This work provides an effective way to integrate Bayesian and
neural-network approaches to cognition.Comment: To be published in Psychological Revie