40 research outputs found
Dynamic Neuro-Symbolic Knowledge Graph Construction for Zero-shot Commonsense Question Answering
Understanding narratives requires reasoning about implicit world knowledge
related to the causes, effects, and states of situations described in text. At
the core of this challenge is how to access contextually relevant knowledge on
demand and reason over it.
In this paper, we present initial studies toward zero-shot commonsense
question answering by formulating the task as inference over dynamically
generated commonsense knowledge graphs. In contrast to previous studies for
knowledge integration that rely on retrieval of existing knowledge from static
knowledge graphs, our study requires commonsense knowledge integration where
contextually relevant knowledge is often not present in existing knowledge
bases. Therefore, we present a novel approach that generates
contextually-relevant symbolic knowledge structures on demand using generative
neural commonsense knowledge models.
Empirical results on two datasets demonstrate the efficacy of our
neuro-symbolic approach for dynamically constructing knowledge graphs for
reasoning. Our approach achieves significant performance boosts over pretrained
language models and vanilla knowledge models, all while providing interpretable
reasoning paths for its predictions
Mitigating Label Biases for In-context Learning
Various design settings for in-context learning (ICL), such as the choice and
order of the in-context examples, can bias the model's predictions. While many
studies discuss these design choices, there have been few systematic
investigations into categorizing them and mitigating their impact. In this
work, we define a typology for three types of label biases in ICL for text
classification: vanilla-label bias, context-label bias, and domain-label bias
(which we conceptualize and detect for the first time). Our analysis
demonstrates that prior label bias calibration methods fall short of addressing
all three types of biases. Specifically, domain-label bias restricts LLMs to
random-level performance on many tasks regardless of the choice of in-context
examples. To mitigate the effect of these biases, we propose a simple bias
calibration method that estimates a language model's label bias using random
in-domain words from the task corpus. After controlling for this estimated bias
when making predictions, our novel domain-context calibration significantly
improves the ICL performance of GPT-J and GPT-3 on a wide range of tasks. The
gain is substantial on tasks with large domain-label bias (up to 37% in
Macro-F1). Furthermore, our results generalize to models with different scales,
pretraining methods, and manually-designed task instructions, showing the
prevalence of label biases in ICL.Comment: Accepted to ACL 202
Commonsense Knowledge Base Completion with Structural and Semantic Context
Automatic KB completion for commonsense knowledge graphs (e.g., ATOMIC and
ConceptNet) poses unique challenges compared to the much studied conventional
knowledge bases (e.g., Freebase). Commonsense knowledge graphs use free-form
text to represent nodes, resulting in orders of magnitude more nodes compared
to conventional KBs (18x more nodes in ATOMIC compared to Freebase
(FB15K-237)). Importantly, this implies significantly sparser graph structures
- a major challenge for existing KB completion methods that assume densely
connected graphs over a relatively smaller set of nodes. In this paper, we
present novel KB completion models that can address these challenges by
exploiting the structural and semantic context of nodes. Specifically, we
investigate two key ideas: (1) learning from local graph structure, using graph
convolutional networks and automatic graph densification and (2) transfer
learning from pre-trained language models to knowledge graphs for enhanced
contextual representation of knowledge. We describe our method to incorporate
information from both these sources in a joint model and provide the first
empirical results for KB completion on ATOMIC and evaluation with ranking
metrics on ConceptNet. Our results demonstrate the effectiveness of language
model representations in boosting link prediction performance and the
advantages of learning from local graph structure (+1.5 points in MRR for
ConceptNet) when training on subgraphs for computational efficiency. Further
analysis on model predictions shines light on the types of commonsense
knowledge that language models capture well.Comment: AAAI 202
Breaking the Language Barrier: Improving Cross-Lingual Reasoning with Structured Self-Attention
In this work, we study whether multilingual language models (MultiLMs) can
transfer logical reasoning abilities to other languages when they are
fine-tuned for reasoning in a different language. We evaluate the cross-lingual
reasoning abilities of MultiLMs in two schemes: (1) where the language of the
context and the question remain the same in the new languages that are tested
(i.e., the reasoning is still monolingual, but the model must transfer the
learned reasoning ability across languages), and (2) where the language of the
context and the question is different (which we term code-switched reasoning).
On two logical reasoning datasets, RuleTaker and LeapOfThought, we demonstrate
that although MultiLMs can transfer reasoning ability across languages in a
monolingual setting, they struggle to transfer reasoning abilities in a
code-switched setting. Following this observation, we propose a novel attention
mechanism that uses a dedicated set of parameters to encourage cross-lingual
attention in code-switched sequences, which improves the reasoning performance
by up to 14% and 4% on the RuleTaker and LeapOfThought datasets, respectively.Comment: EMNLP 2023 - Finding
CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks
Recent efforts in natural language processing (NLP) commonsense reasoning
research have yielded a considerable number of new datasets and benchmarks.
However, most of these datasets formulate commonsense reasoning challenges in
artificial scenarios that are not reflective of the tasks which real-world NLP
systems are designed to solve. In this work, we present CRoW, a
manually-curated, multi-task benchmark that evaluates the ability of models to
apply commonsense reasoning in the context of six real-world NLP tasks. CRoW is
constructed using a multi-stage data collection pipeline that rewrites examples
from existing datasets using commonsense-violating perturbations. We use CRoW
to study how NLP systems perform across different dimensions of commonsense
knowledge, such as physical, temporal, and social reasoning. We find a
significant performance gap when NLP systems are evaluated on CRoW compared to
humans, showcasing that commonsense reasoning is far from being solved in
real-world task settings. We make our dataset and leaderboard available to the
research community at https://github.com/mismayil/crow.Comment: 37 pages, camera-ready for EMNLP 202
QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering
The problem of answering questions using knowledge from pre-trained language
models (LMs) and knowledge graphs (KGs) presents two challenges: given a QA
context (question and answer choice), methods need to (i) identify relevant
knowledge from large KGs, and (ii) perform joint reasoning over the QA context
and KG. In this work, we propose a new model, QA-GNN, which addresses the above
challenges through two key innovations: (i) relevance scoring, where we use LMs
to estimate the importance of KG nodes relative to the given QA context, and
(ii) joint reasoning, where we connect the QA context and KG to form a joint
graph, and mutually update their representations through graph neural networks.
We evaluate our model on QA benchmarks in the commonsense (CommonsenseQA,
OpenBookQA) and biomedical (MedQA-USMLE) domains. QA-GNN outperforms existing
LM and LM+KG models, and exhibits capabilities to perform interpretable and
structured reasoning, e.g., correctly handling negation in questions.Comment: NAACL 2021. Code & data available at
https://github.com/michiyasunaga/qagn
DISCO: Distilling Phrasal Counterfactuals with Large Language Models
Models trained with counterfactually augmented data learn representations of
the causal structure of tasks, enabling robust generalization. However,
high-quality counterfactual data is scarce for most tasks and not easily
generated at scale. When crowdsourced, such data is typically limited in scale
and diversity; when generated using supervised methods, it is computationally
expensive to extend to new counterfactual dimensions. In this work, we
introduce DISCO (DIStilled COunterfactual Data), a new method for automatically
generating high quality counterfactual data at scale. DISCO engineers prompts
to generate phrasal perturbations with a large general language model. Then, a
task-specific teacher model filters these generations to distill high-quality
counterfactual data. While task-agnostic, we apply our pipeline to the task of
natural language inference (NLI) and find that on challenging evaluations such
as the NLI stress test, comparatively smaller student models trained with DISCO
generated counterfactuals are more robust (6% absolute) and generalize better
across distributions (2%) compared to models trained without data augmentation.
Furthermore, DISCO augmented models are 10% more consistent between
counterfactual pairs on three evaluation sets, demonstrating that DISCO
augmentation enables models to more reliably learn causal representations. Our
repository is available at: https://github.com/eric11eca/discoComment: ACL 2023 camera read