25,089 research outputs found
DeepProbLog: Neural Probabilistic Logic Programming
We introduce DeepProbLog, a probabilistic logic programming language that
incorporates deep learning by means of neural predicates. We show how existing
inference and learning techniques can be adapted for the new language. Our
experiments demonstrate that DeepProbLog supports both symbolic and subsymbolic
representations and inference, 1) program induction, 2) probabilistic (logic)
programming, and 3) (deep) learning from examples. To the best of our
knowledge, this work is the first to propose a framework where general-purpose
neural networks and expressive probabilistic-logical modeling and reasoning are
integrated in a way that exploits the full expressiveness and strengths of both
worlds and can be trained end-to-end based on examples.Comment: Accepted for spotlight at NeurIPS 201
Relational Neural Machines
Deep learning has been shown to achieve impressive results in several tasks
where a large amount of training data is available. However, deep learning
solely focuses on the accuracy of the predictions, neglecting the reasoning
process leading to a decision, which is a major issue in life-critical
applications. Probabilistic logic reasoning allows to exploit both statistical
regularities and specific domain expertise to perform reasoning under
uncertainty, but its scalability and brittle integration with the layers
processing the sensory data have greatly limited its applications. For these
reasons, combining deep architectures and probabilistic logic reasoning is a
fundamental goal towards the development of intelligent agents operating in
complex environments. This paper presents Relational Neural Machines, a novel
framework allowing to jointly train the parameters of the learners and of a
First--Order Logic based reasoner. A Relational Neural Machine is able to
recover both classical learning from supervised data in case of pure
sub-symbolic learning, and Markov Logic Networks in case of pure symbolic
reasoning, while allowing to jointly train and perform inference in hybrid
learning tasks. Proper algorithmic solutions are devised to make learning and
inference tractable in large-scale problems. The experiments show promising
results in different relational tasks
Optimisation in Neurosymbolic Learning Systems
In the last few years, Artificial Intelligence (AI) has reached the public consciousness through high-profile applications such as chatbots, image generators, speech synthesis and transcription. These are all due to the success of deep learning: Machine learning algorithms that learn tasks from massive amounts of data. The neural network models used in deep learning involve many parameters, often in the order of billions. These models often fail on tasks that computers are traditionally very good at, like calculating arithmetic expressions, reasoning about many different pieces of information, planning and scheduling complex systems, and retrieving information from a database. These tasks are traditionally solved using symbolic methods in AI based on logic and formal reasoning. Neurosymbolic AI instead aims to integrate deep learning with symbolic AI. This integration has many promises, such as decreasing the amount of data required to train the neural networks, improving the explainability and interpretability of answers given by models and verifying the correctness of trained systems. We mainly study neurosymbolic learning, where we have, in addition to data, background knowledge expressed using symbolic languages. How do we connect the symbolic and neural components to communicate this knowledge to the neural networks? We consider two answers: Fuzzy and probabilistic reasoning. Fuzzy reasoning studies degrees of truth. A person can be very or somewhat tall: Tallness is not a binary concept. Instead, probabilistic reasoning studies the probability that something is true or will happen. A coin has a 0.5 probability of landing heads. We never say it landed on "somewhat heads". What happens when we use fuzzy (part I) or probabilistic (part II) approaches to neurosymbolic learning? Moreover, do these approaches use the background knowledge we expect them to? Our first research question studies how different forms of fuzzy reasoning combine with learning. We find surprising results like a connection to the Raven paradox, which states that we confirm "ravens are black" when we observe a green apple. In this study, we gave our neural network a training objective created from the background knowledge. However, we did not use the background knowledge when we deployed our models after training. In our second research question, we studied how to use background knowledge in deployed models. To this end, we developed a new neural network layer based on fuzzy reasoning. The remaining research questions study probabilistic approaches to neurosymbolic learning. Probabilistic reasoning is a natural fit for neural networks, which we usually train to be probabilistic. However, probabilistic approaches come at a cost: They are expensive to compute and do not scale well to large tasks. In our third research question, we study how to connect probabilistic reasoning with neural networks by sampling to estimate averages. Sampling circumvents computing reasoning outcomes for all input combinations. In the fourth and final research question, we study scaling probabilistic neurosymbolic learning to much larger problems than possible before. Our insight is to train a neural network to predict the result of probabilistic reasoning. We perform this training process with just the background knowledge: We do not collect data. How is this related to optimisation? All research questions are related to optimisation problems. Within neurosymbolic learning, optimisation with popular methods like gradient descent undertake a form of reasoning. There is ample opportunity to study how this optimisation perspective improves our neurosymbolic learning methods. We hope this dissertation provides some of the answers needed to make practical neurosymbolic learning a reality: Where practitioners provide both data and knowledge that the neurosymbolic learning methods use as efficiently as possible to train the next generation of neural networks
Logical Activation Functions: Logit-space equivalents of Probabilistic Boolean Operators
The choice of activation functions and their motivation is a long-standing
issue within the neural network community. Neuronal representations within
artificial neural networks are commonly understood as logits, representing the
log-odds score of presence of features within the stimulus. We derive
logit-space operators equivalent to probabilistic Boolean logic-gates AND, OR,
and XNOR for independent probabilities. Such theories are important to
formalize more complex dendritic operations in real neurons, and these
operations can be used as activation functions within a neural network,
introducing probabilistic Boolean-logic as the core operation of the neural
network. Since these functions involve taking multiple exponents and
logarithms, they are computationally expensive and not well suited to be
directly used within neural networks. Consequently, we construct efficient
approximations named (the AND operator Approximate for
Independent Logits), , and ,
which utilize only comparison and addition operations, have well-behaved
gradients, and can be deployed as activation functions in neural networks. Like
MaxOut, and are generalizations
of ReLU to two-dimensions. While our primary aim is to formalize dendritic
computations within a logit-space probabilistic-Boolean framework, we deploy
these new activation functions, both in isolation and in conjunction to
demonstrate their effectiveness on a variety of tasks including image
classification, transfer learning, abstract reasoning, and compositional
zero-shot learning
DeepProbLog: neural probabilistic logic programming
We introduce DeepProbLog, a probabilistic logic programming language that incorporates deep learning by means of neural predicates. We show how existing inference and learning techniques can be adapted for the new language. Our experiments demonstrate that DeepProbLog supports (i) both symbolic and subsymbolic representations and inference, (ii) program induction, (iii) probabilistic (logic) programming, and (iv) (deep) learning from examples. To the best of our knowledge, this work is the first to propose a framework where general-purpose neural networks and expressive probabilistic-logical modeling and reasoning are integrated in a way that exploits the full expressiveness and strengths of both worlds and can be trained end-to-end based on examples
- …