22 research outputs found
Learning Reasoning Strategies in End-to-End Differentiable Proving
Attempts to render deep learning models interpretable, data-efficient, and
robust have seen some success through hybridisation with rule-based systems,
for example, in Neural Theorem Provers (NTPs). These neuro-symbolic models can
induce interpretable rules and learn representations from data via
back-propagation, while providing logical explanations for their predictions.
However, they are restricted by their computational complexity, as they need to
consider all possible proof paths for explaining a goal, thus rendering them
unfit for large-scale applications. We present Conditional Theorem Provers
(CTPs), an extension to NTPs that learns an optimal rule selection strategy via
gradient-based optimisation. We show that CTPs are scalable and yield
state-of-the-art results on the CLUTRR dataset, which tests systematic
generalisation of neural models by learning to reason over smaller graphs and
evaluating on larger ones. Finally, CTPs show better link prediction results on
standard benchmarks in comparison with other neural-symbolic models, while
being explainable. All source code and datasets are available online, at
https://github.com/uclnlp/ctp.Comment: Proceedings of the 37th International Conference on Machine Learning
(ICML 2020
Measuring Systematic Generalization in Neural Proof Generation with Transformers
We are interested in understanding how well Transformer language models
(TLMs) can perform reasoning tasks when trained on knowledge encoded in the
form of natural language. We investigate systematic generalization abilities on
an inductive logical reasoning task in natural language, which involves
reasoning over relationships between entities grounded in first-order logical
proofs. Specifically, we perform soft theorem-proving by leveraging TLMs to
generate logical proofs represented in natural language. We systematically test
proof generation capabilities, along with inference capabilities leveraging the
generated proofs. We observe length-generalization issues in proof generation
and inference when evaluated on longer-than-trained sequences. However, we
observe TLMs improve their generalization performance after being exposed to
longer, exhaustive proofs. In addition, we discover that TLMs are able to
generalize better using backward-chaining proofs compared to their
forward-chaining counterparts, while they find it easier to generate forward
chaining proofs. We observe that models that are not trained to generate proofs
are better at generalizing to problems based on longer proofs. This result
suggests that Transformers have efficient, yet not interpretable reasoning
strategies internally. These results also highlight the systematic
generalization issues in TLMs in the context of logical reasoning, and we
believe this work will motivate deeper inspection of their underlying reasoning
strategies.Comment: NeurIPS 2020; 17 pages; 9 figures; 6 table
Complex Query Answering on Eventuality Knowledge Graph with Implicit Logical Constraints
Querying incomplete knowledge graphs (KGs) using deep learning approaches can
naturally leverage the reasoning and generalization ability to learn to infer
better answers. Traditional neural complex query answering (CQA) approaches
mostly work on entity-centric KGs. However, in the real world, we also need to
make logical inferences about events, states, and activities (i.e.,
eventualities or situations) to push learning systems from System I to System
II, as proposed by Yoshua Bengio. Querying logically from an
EVentuality-centric KG (EVKG) can naturally provide references to such kind of
intuitive and logical inference. Thus, in this paper, we propose a new
framework to leverage neural methods to answer complex logical queries based on
an EVKG, which can satisfy not only traditional first-order logic constraints
but also implicit logical constraints over eventualities concerning their
occurrences and orders. For instance, if we know that ``Food is bad'' happens
before ``PersonX adds soy sauce,'' then ``PersonX adds soy sauce'' is unlikely
to be the cause of ``Food is bad'' due to implicit temporal constraint. To
facilitate consistent reasoning on EVKGs, we propose Complex Eventuality Query
Answering (CEQA), a more rigorous definition of CQA that considers the implicit
logical constraints governing the temporal order and occurrence of
eventualities. In this manner, we propose to leverage theorem provers for
constructing benchmark datasets to ensure the answers satisfy implicit logical
constraints. We also propose a Memory-Enhanced Query Encoding (MEQE) approach
to significantly improve the performance of state-of-the-art neural query
encoders on the CEQA task