15 research outputs found
A Bayesian Unification of Self-Supervised Clustering and Energy-Based Models
Self-supervised learning is a popular and powerful method for utilizing large
amounts of unlabeled data, for which a wide variety of training objectives have
been proposed in the literature. In this study, we perform a Bayesian analysis
of state-of-the-art self-supervised learning objectives, elucidating the
underlying probabilistic graphical models in each class and presenting a
standardized methodology for their derivation from first principles. The
analysis also indicates a natural means of integrating self-supervised learning
with likelihood-based generative models. We instantiate this concept within the
realm of cluster-based self-supervised learning and energy models, introducing
a novel lower bound which is proven to reliably penalize the most important
failure modes. Furthermore, this newly proposed lower bound enables the
training of a standard backbone architecture without the necessity for
asymmetric elements such as stop gradients, momentum encoders, or specialized
clustering layers - typically introduced to avoid learning trivial solutions.
Our theoretical findings are substantiated through experiments on synthetic and
real-world data, including SVHN, CIFAR10, and CIFAR100, thus showing that our
objective function allows to outperform existing self-supervised learning
strategies in terms of clustering, generation and out-of-distribution detection
performance by a wide margin. We also demonstrate that GEDI can be integrated
into a neuro-symbolic framework to mitigate the reasoning shortcut problem and
to learn higher quality symbolic representations thanks to the enhanced
classification performance.Comment: Changes from previous version: added mean and standard deviations in
experiments. Integral version of workshop paper arXiv:2309.15420. Improved
GEDI version (from two stages to single stage training) arxiv:2212.1342
From Statistical Relational to Neurosymbolic Artificial Intelligence: a Survey
This survey explores the integration of learning and reasoning in two
different fields of artificial intelligence: neurosymbolic and statistical
relational artificial intelligence. Neurosymbolic artificial intelligence
(NeSy) studies the integration of symbolic reasoning and neural networks, while
statistical relational artificial intelligence (StarAI) focuses on integrating
logic with probabilistic graphical models. This survey identifies seven shared
dimensions between these two subfields of AI. These dimensions can be used to
characterize different NeSy and StarAI systems. They are concerned with (1) the
approach to logical inference, whether model or proof-based; (2) the syntax of
the used logical theories; (3) the logical semantics of the systems and their
extensions to facilitate learning; (4) the scope of learning, encompassing
either parameter or structure learning; (5) the presence of symbolic and
subsymbolic representations; (6) the degree to which systems capture the
original logic, probabilistic, and neural paradigms; and (7) the classes of
learning tasks the systems are applied to. By positioning various NeSy and
StarAI systems along these dimensions and pointing out similarities and
differences between them, this survey contributes fundamental concepts for
understanding the integration of learning and reasoning.Comment: To appear in Artificial Intelligence. Shorter version at IJCAI 2020
survey track, https://www.ijcai.org/proceedings/2020/0688.pd
DeepProbLog: Neural Probabilistic Logic Programming
We introduce DeepProbLog, a probabilistic logic programming language that
incorporates deep learning by means of neural predicates. We show how existing
inference and learning techniques can be adapted for the new language. Our
experiments demonstrate that DeepProbLog supports both symbolic and subsymbolic
representations and inference, 1) program induction, 2) probabilistic (logic)
programming, and 3) (deep) learning from examples. To the best of our
knowledge, this work is the first to propose a framework where general-purpose
neural networks and expressive probabilistic-logical modeling and reasoning are
integrated in a way that exploits the full expressiveness and strengths of both
worlds and can be trained end-to-end based on examples.Comment: Accepted for spotlight at NeurIPS 201
Neural Probabilistic Logic Programming in Discrete-Continuous Domains
Neural-symbolic AI (NeSy) allows neural networks to exploit symbolic
background knowledge in the form of logic. It has been shown to aid learning in
the limited data regime and to facilitate inference on out-of-distribution
data. Probabilistic NeSy focuses on integrating neural networks with both logic
and probability theory, which additionally allows learning under uncertainty. A
major limitation of current probabilistic NeSy systems, such as DeepProbLog, is
their restriction to finite probability distributions, i.e., discrete random
variables. In contrast, deep probabilistic programming (DPP) excels in
modelling and optimising continuous probability distributions. Hence, we
introduce DeepSeaProbLog, a neural probabilistic logic programming language
that incorporates DPP techniques into NeSy. Doing so results in the support of
inference and learning of both discrete and continuous probability
distributions under logical constraints. Our main contributions are 1) the
semantics of DeepSeaProbLog and its corresponding inference algorithm, 2) a
proven asymptotically unbiased learning algorithm, and 3) a series of
experiments that illustrate the versatility of our approach.Comment: 27 pages, 9 figure
DeepProbLog: neural probabilistic logic programming
We introduce DeepProbLog, a probabilistic logic programming language that incorporates deep learning by means of neural predicates. We show how existing inference and learning techniques can be adapted for the new language. Our experiments demonstrate that DeepProbLog supports (i) both symbolic and subsymbolic representations and inference, (ii) program induction, (iii) probabilistic (logic) programming, and (iv) (deep) learning from examples. To the best of our knowledge, this work is the first to propose a framework where general-purpose neural networks and expressive probabilistic-logical modeling and reasoning are integrated in a way that exploits the full expressiveness and strengths of both worlds and can be trained end-to-end based on examples
Three Modern Roles for Logic in AI
We consider three modern roles for logic in artificial intelligence, which
are based on the theory of tractable Boolean circuits: (1) logic as a basis for
computation, (2) logic for learning from a combination of data and knowledge,
and (3) logic for reasoning about the behavior of machine learning systems.Comment: To be published in PODS 202
DeepStochLog: Neural Stochastic Logic Programming
Recent advances in neural-symbolic learning, such as DeepProbLog, extend probabilistic logic programs with neural predicates. Like graphical models, these probabilistic logic programs define a probability distribution over possible worlds, for which inference is computationally hard. We propose DeepStochLog, an alternative neural-symbolic framework based on stochastic definite clause grammars, a kind of stochastic logic program. More specifically, we introduce neural grammar rules into stochastic definite clause grammars to create a framework that can be trained end-to-end. We show that inference and learning in neural stochastic logic programming scale much better than for neural probabilistic logic programs. Furthermore, the experimental evaluation shows that DeepStochLog achieves state-of-the-art results on challenging neural-symbolic learning tasks