211,415 research outputs found
Enhancing Logical Reasoning of Large Language Models through Logic-Driven Data Augmentation
Combining large language models with logical reasoning enhance their capacity
to address problems in a robust and reliable manner. Nevertheless, the
intricate nature of logical reasoning poses challenges to gathering reliable
data from web for building comprehensive training datasets, subsequently
affecting the performance on downstream tasks. To address this, we introduce a
novel logic-driven data augmentation approach, AMR-LDA. AMR-LDA converts the
original text into an Abstract Meaning Representation (AMR) graph, a structured
semantic representation that encapsulates the logic structure of the sentence,
upon which operations are performed to generate logically modified AMR graphs.
The modified AMR graphs are subsequently converted back into texts to create
augmented data. Notably, our methodology is architecture-agnostic and enhances
generative large language models, such as GPT-3.5 and GPT-4, through prompt
augmentation, and fine-tuning discriminative large language models through
contrastive learning with logic-driven data augmentation. Empirical evidence
underscores the efficacy of our proposed method with improvement in performance
across seven downstream tasks, such as logical reasoning reading comprehension,
textual entailment, and natural language inference. Furthermore, our method
ranked first on the ReClor leaderboard
\url{https://eval.ai/web/challenges/challenge-page/503/leaderboard/1347}. The
source code and data are publicly available
\url{https://github.com/Strong-AI-Lab/Logical-Equivalence-driven-AMR-Data-Augmentation-for-Representation-Learning}.Comment: Accepted for oral presentation at the LLM@IJCAI 2023 non-archival
symposiu
A Systematic Evaluation of Large Language Models on Out-of-Distribution Logical Reasoning Tasks
Large language models (LLMs), such as GPT-3.5 and GPT-4, have greatly
advanced the performance of artificial systems on various natural language
processing tasks to human-like levels. However, their generalisation and
robustness to perform logical reasoning remain under-evaluated. To probe this
ability, we propose three new logical reasoning datasets named "ReClor-plus",
"LogiQA-plus" and "LogiQAv2-plus", each featuring three subsets: the first with
randomly shuffled options, the second with the correct choices replaced by
"none of the other options are correct", and a combination of the previous two
subsets. We carry out experiments on these datasets with both discriminative
and generative LLMs and show that these simple tricks greatly hinder the
performance of the language models. Despite their superior performance on the
original publicly available datasets, we find that all models struggle to
answer our newly constructed datasets. We show that introducing task variations
by perturbing a sizable training set can markedly improve the model's
generalisation and robustness in logical reasoning tasks. Moreover, applying
logic-driven data augmentation for fine-tuning, combined with prompting can
enhance the generalisation performance of both discriminative large language
models and generative large language models. These results offer insights into
assessing and improving the generalisation and robustness of large language
models for logical reasoning tasks. We make our source code and data publicly
available
\url{https://github.com/Strong-AI-Lab/Logical-and-abstract-reasoning}.Comment: Accepted for oral presentation at the LLM@IJCAI 2023 non-archival
symposiu
Lesions to Polar/Orbital Prefrontal Cortex Selectively Impair Reasoning about Emotional Material
While it is widely accepted that lesions to orbital prefrontal cortex lead to emotion related disruptions and poor decision-making, there is very little patient data on this issue involving actual logical reasoning tasks. We tested patients with circumscribed, focal lesions largely confined to polar/orbital prefrontal cortex (BA 10 and 11) (N=17) on logical reasoning tasks involving neutral and emotional content, and compared their performance to that of an age and education-matched normal control group (N=22) and a posterior lesion control group (N=24). Our results revealed a significant group by content interaction driven by a selective impairment in the polar/orbital prefrontal cortex group compared to healthy normal controls and to the parietal patient group, in the emotional content reasoning trials. Subsequent analyses of congruent and incongruent reasoning trials indicated that this impairment was driven by the poor performance of patients with polar/orbital lesions in the incongruent trials. We conclude that the polar/orbital prefrontal cortex plays a critical role in filtering emotionally charged content from the material before it is passed on to the reasoning system in lateral/dorsal regions of prefrontal cortex. Where unfiltered content is passed to the reasoning engine, either as a result of pathology (as in the case of our patients) or as a result of individual differences, reasoning performance suffers
Logic tensor networks for semantic image interpretation
Semantic Image Interpretation (SII) is the task of extracting structured semantic descriptions from images. It is widely agreed that the combined use of visual data and background knowledge is of great importance for SII. Recently, Statistical Relational Learning (SRL) approaches have been developed for reasoning under uncertainty and learning in the presence of data and rich knowledge. Logic Tensor Networks (LTNs) are a SRL framework which integrates neural networks with first-order fuzzy logic to allow (i) efficient learning from noisy data in the presence of logical constraints, and (ii) reasoning with logical formulas describing general properties of the data. In this paper, we develop and apply LTNs to two of the main tasks of SII, namely, the classification of an image's bounding boxes and the detection of the relevant part-of relations between objects. To the best of our knowledge, this is the first successful application of SRL to such SII tasks. The proposed approach is evaluated on a standard image processing benchmark. Experiments show that background knowledge in the form of logical constraints can improve the performance of purely data-driven approaches, including the state-of-theart Fast Region-based Convolutional Neural Networks (Fast R-CNN). Moreover, we show that the use of logical background knowledge adds robustness to the learning system when errors are present in the labels of the training data
Invalid Logic, Equivalent Gains: The Bizarreness of Reasoning in Language Model Prompting
Language models can be prompted to reason through problems in a manner that
significantly improves performance. However, \textit{why} such prompting
improves performance is unclear. Recent work showed that using logically
\textit{invalid} Chain-of-Thought (CoT) prompting improves performance almost
as much as logically \textit{valid} CoT prompting, and that editing CoT prompts
to replace problem-specific information with abstract information or
out-of-distribution information typically doesn't harm performance. Critics
have responded that these findings are based on too few and too easily solved
tasks to draw meaningful conclusions. To resolve this dispute, we test whether
logically invalid CoT prompts offer the same level of performance gains as
logically valid prompts on the hardest tasks in the BIG-Bench benchmark, termed
BIG-Bench Hard (BBH). We find that the logically \textit{invalid} reasoning
prompts do indeed achieve similar performance gains on BBH tasks as logically
valid reasoning prompts. We also discover that some CoT prompts used by
previous works contain logical errors. This suggests that covariates beyond
logically valid reasoning are responsible for performance improvements.Comment: ICML 2023 Workshop: Knowledge and Logical Reasoning in the Era of
Data-driven Learnin
Improving Certified Robustness via Statistical Learning with Logical Reasoning
Intensive algorithmic efforts have been made to enable the rapid improvements
of certificated robustness for complex ML models recently. However, current
robustness certification methods are only able to certify under a limited
perturbation radius. Given that existing pure data-driven statistical
approaches have reached a bottleneck, in this paper, we propose to integrate
statistical ML models with knowledge (expressed as logical rules) as a
reasoning component using Markov logic networks (MLN, so as to further improve
the overall certified robustness. This opens new research questions about
certifying the robustness of such a paradigm, especially the reasoning
component (e.g., MLN). As the first step towards understanding these questions,
we first prove that the computational complexity of certifying the robustness
of MLN is #P-hard. Guided by this hardness result, we then derive the first
certified robustness bound for MLN by carefully analyzing different model
regimes. Finally, we conduct extensive experiments on five datasets including
both high-dimensional images and natural language texts, and we show that the
certified robustness with knowledge-based logical reasoning indeed
significantly outperforms that of the state-of-the-art
LOGICSEG: Parsing Visual Semantics with Neural Logic Learning and Reasoning
Current high-performance semantic segmentation models are purely data-driven
sub-symbolic approaches and blind to the structured nature of the visual world.
This is in stark contrast to human cognition which abstracts visual perceptions
at multiple levels and conducts symbolic reasoning with such structured
abstraction. To fill these fundamental gaps, we devise LOGICSEG, a holistic
visual semantic parser that integrates neural inductive learning and logic
reasoning with both rich data and symbolic knowledge. In particular, the
semantic concepts of interest are structured as a hierarchy, from which a set
of constraints are derived for describing the symbolic relations and formalized
as first-order logic rules. After fuzzy logic-based continuous relaxation,
logical formulae are grounded onto data and neural computational graphs, hence
enabling logic-induced network training. During inference, logical constraints
are packaged into an iterative process and injected into the network in a form
of several matrix multiplications, so as to achieve hierarchy-coherent
prediction with logic reasoning. These designs together make LOGICSEG a general
and compact neural-logic machine that is readily integrated into existing
segmentation models. Extensive experiments over four datasets with various
segmentation models and backbones verify the effectiveness and generality of
LOGICSEG. We believe this study opens a new avenue for visual semantic parsing.Comment: ICCV 2023 (Oral). Code: https://github.com/lingorX/LogicSeg
- …