68 research outputs found
Adversarial Sets for Regularising Neural Link Predictors
In adversarial training, a set of models learn together by pursuing competing
goals, usually defined on single data instances. However, in relational
learning and other non-i.i.d domains, goals can also be defined over sets of
instances. For example, a link predictor for the is-a relation needs to be
consistent with the transitivity property: if is-a(x_1, x_2) and is-a(x_2, x_3)
hold, is-a(x_1, x_3) needs to hold as well. Here we use such assumptions for
deriving an inconsistency loss, measuring the degree to which the model
violates the assumptions on an adversarially-generated set of examples. The
training objective is defined as a minimax problem, where an adversary finds
the most offending adversarial examples by maximising the inconsistency loss,
and the model is trained by jointly minimising a supervised loss and the
inconsistency loss on the adversarial examples. This yields the first method
that can use function-free Horn clauses (as in Datalog) to regularise any
neural link predictor, with complexity independent of the domain size. We show
that for several link prediction models, the optimisation problem faced by the
adversary has efficient closed-form solutions. Experiments on link prediction
benchmarks indicate that given suitable prior knowledge, our method can
significantly improve neural link predictors on all relevant metrics.Comment: Proceedings of the 33rd Conference on Uncertainty in Artificial
Intelligence (UAI), 201
Adversarially Regularising Neural NLI Models to Integrate Logical Background Knowledge
Adversarial examples are inputs to machine learning models designed to cause
the model to make a mistake. They are useful for understanding the shortcomings
of machine learning models, interpreting their results, and for regularisation.
In NLP, however, most example generation strategies produce input text by using
known, pre-specified semantic transformations, requiring significant manual
effort and in-depth understanding of the problem and domain. In this paper, we
investigate the problem of automatically generating adversarial examples that
violate a set of given First-Order Logic constraints in Natural Language
Inference (NLI). We reduce the problem of identifying such adversarial examples
to a combinatorial optimisation problem, by maximising a quantity measuring the
degree of violation of such constraints and by using a language model for
generating linguistically-plausible examples. Furthermore, we propose a method
for adversarially regularising neural NLI models for incorporating background
knowledge. Our results show that, while the proposed method does not always
improve results on the SNLI and MultiNLI datasets, it significantly and
consistently increases the predictive accuracy on adversarially-crafted
datasets -- up to a 79.6% relative improvement -- while drastically reducing
the number of background knowledge violations. Furthermore, we show that
adversarial examples transfer among model architectures, and that the proposed
adversarial training procedure improves the robustness of NLI models to
adversarial examples.Comment: Accepted at the SIGNLL Conference on Computational Natural Language
Learning (CoNLL 2018
Learning Reasoning Strategies in End-to-End Differentiable Proving
Attempts to render deep learning models interpretable, data-efficient, and
robust have seen some success through hybridisation with rule-based systems,
for example, in Neural Theorem Provers (NTPs). These neuro-symbolic models can
induce interpretable rules and learn representations from data via
back-propagation, while providing logical explanations for their predictions.
However, they are restricted by their computational complexity, as they need to
consider all possible proof paths for explaining a goal, thus rendering them
unfit for large-scale applications. We present Conditional Theorem Provers
(CTPs), an extension to NTPs that learns an optimal rule selection strategy via
gradient-based optimisation. We show that CTPs are scalable and yield
state-of-the-art results on the CLUTRR dataset, which tests systematic
generalisation of neural models by learning to reason over smaller graphs and
evaluating on larger ones. Finally, CTPs show better link prediction results on
standard benchmarks in comparison with other neural-symbolic models, while
being explainable. All source code and datasets are available online, at
https://github.com/uclnlp/ctp.Comment: Proceedings of the 37th International Conference on Machine Learning
(ICML 2020
Constraint-Based Visual Generation
In the last few years the systematic adoption of deep learning to visual
generation has produced impressive results that, amongst others, definitely
benefit from the massive exploration of convolutional architectures. In this
paper, we propose a general approach to visual generation that combines
learning capabilities with logic descriptions of the target to be generated.
The process of generation is regarded as a constrained satisfaction problem,
where the constraints describe a set of properties that characterize the
target. Interestingly, the constraints can also involve logic variables, while
all of them are converted into real-valued functions by means of the t-norm
theory. We use deep architectures to model the involved variables, and propose
a computational scheme where the learning process carries out a satisfaction of
the constraints. We propose some examples in which the theory can naturally be
used, including the modeling of GAN and auto-encoders, and report promising
results in problems with the generation of handwritten characters and face
transformations
- …