218 research outputs found
Commonsense Reasoning for Conversational AI: A Survey of the State of the Art
Large, transformer-based pretrained language models like BERT, GPT, and T5
have demonstrated a deep understanding of contextual semantics and language
syntax. Their success has enabled significant advances in conversational AI,
including the development of open-dialogue systems capable of coherent, salient
conversations which can answer questions, chat casually, and complete tasks.
However, state-of-the-art models still struggle with tasks that involve higher
levels of reasoning - including commonsense reasoning that humans find trivial.
This paper presents a survey of recent conversational AI research focused on
commonsense reasoning. The paper lists relevant training datasets and describes
the primary approaches to include commonsense in conversational AI. The paper
also discusses benchmarks used for evaluating commonsense in conversational AI
problems. Finally, the paper presents preliminary observations of the limited
commonsense capabilities of two state-of-the-art open dialogue models,
BlenderBot3 and LaMDA, and its negative effect on natural interactions. These
observations further motivate research on commonsense reasoning in
conversational AI.Comment: Accepted to Workshop on Knowledge Augmented Methods for Natural
Language Processing, in conjunction with AAAI 202
Reliable Natural Language Understanding with Large Language Models and Answer Set Programming
Humans understand language by extracting information (meaning) from
sentences, combining it with existing commonsense knowledge, and then
performing reasoning to draw conclusions. While large language models (LLMs)
such as GPT-3 and ChatGPT are able to leverage patterns in the text to solve a
variety of NLP tasks, they fall short in problems that require reasoning. They
also cannot reliably explain the answers generated for a given question. In
order to emulate humans better, we propose STAR, a framework that combines LLMs
with Answer Set Programming (ASP). We show how LLMs can be used to effectively
extract knowledge -- represented as predicates -- from language. Goal-directed
ASP is then employed to reliably reason over this knowledge. We apply the STAR
framework to three different NLU tasks requiring reasoning: qualitative
reasoning, mathematical reasoning, and goal-directed conversation. Our
experiments reveal that STAR is able to bridge the gap of reasoning in NLU
tasks, leading to significant performance improvements, especially for smaller
LLMs, i.e., LLMs with a smaller number of parameters. NLU applications
developed using the STAR framework are also explainable: along with the
predicates generated, a justification in the form of a proof tree can be
produced for a given output.Comment: In Proceedings ICLP 2023, arXiv:2308.1489
Is Neuro-Symbolic AI Meeting its Promise in Natural Language Processing? A Structured Review
Advocates for Neuro-Symbolic Artificial Intelligence (NeSy) assert that
combining deep learning with symbolic reasoning will lead to stronger AI than
either paradigm on its own. As successful as deep learning has been, it is
generally accepted that even our best deep learning systems are not very good
at abstract reasoning. And since reasoning is inextricably linked to language,
it makes intuitive sense that Natural Language Processing (NLP), would be a
particularly well-suited candidate for NeSy. We conduct a structured review of
studies implementing NeSy for NLP, with the aim of answering the question of
whether NeSy is indeed meeting its promises: reasoning, out-of-distribution
generalization, interpretability, learning and reasoning from small data, and
transferability to new domains. We examine the impact of knowledge
representation, such as rules and semantic networks, language structure and
relational structure, and whether implicit or explicit reasoning contributes to
higher promise scores. We find that systems where logic is compiled into the
neural network lead to the most NeSy goals being satisfied, while other factors
such as knowledge representation, or type of neural architecture do not exhibit
a clear correlation with goals being met. We find many discrepancies in how
reasoning is defined, specifically in relation to human level reasoning, which
impact decisions about model architectures and drive conclusions which are not
always consistent across studies. Hence we advocate for a more methodical
approach to the application of theories of human reasoning as well as the
development of appropriate benchmarks, which we hope can lead to a better
understanding of progress in the field. We make our data and code available on
github for further analysis.Comment: Surve
ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigation
The ability to accurately locate and navigate to a specific object is a
crucial capability for embodied agents that operate in the real world and
interact with objects to complete tasks. Such object navigation tasks usually
require large-scale training in visual environments with labeled objects, which
generalizes poorly to novel objects in unknown environments. In this work, we
present a novel zero-shot object navigation method, Exploration with Soft
Commonsense constraints (ESC), that transfers commonsense knowledge in
pre-trained models to open-world object navigation without any navigation
experience nor any other training on the visual environments. First, ESC
leverages a pre-trained vision and language model for open-world prompt-based
grounding and a pre-trained commonsense language model for room and object
reasoning. Then ESC converts commonsense knowledge into navigation actions by
modeling it as soft logic predicates for efficient exploration. Extensive
experiments on MP3D, HM3D, and RoboTHOR benchmarks show that our ESC method
improves significantly over baselines, and achieves new state-of-the-art
results for zero-shot object navigation (e.g., 158% relative Success Rate
improvement than CoW on MP3D)
Neuro Symbolic Reasoning for Planning: Counterexample Guided Inductive Synthesis using Large Language Models and Satisfiability Solving
Generative large language models (LLMs) with instruct training such as GPT-4
can follow human-provided instruction prompts and generate human-like responses
to these prompts. Apart from natural language responses, they have also been
found to be effective at generating formal artifacts such as code, plans, and
logical specifications from natural language prompts. Despite their remarkably
improved accuracy, these models are still known to produce factually incorrect
or contextually inappropriate results despite their syntactic coherence - a
phenomenon often referred to as hallucination. This limitation makes it
difficult to use these models to synthesize formal artifacts that are used in
safety-critical applications. Unlike tasks such as text summarization and
question-answering, bugs in code, plan, and other formal artifacts produced by
LLMs can be catastrophic. We posit that we can use the satisfiability modulo
theory (SMT) solvers as deductive reasoning engines to analyze the generated
solutions from the LLMs, produce counterexamples when the solutions are
incorrect, and provide that feedback to the LLMs exploiting the dialog
capability of instruct-trained LLMs. This interaction between inductive LLMs
and deductive SMT solvers can iteratively steer the LLM to generate the correct
response. In our experiments, we use planning over the domain of blocks as our
synthesis task for evaluating our approach. We use GPT-4, GPT3.5 Turbo,
Davinci, Curie, Babbage, and Ada as the LLMs and Z3 as the SMT solver. Our
method allows the user to communicate the planning problem in natural language;
even the formulation of queries to SMT solvers is automatically generated from
natural language. Thus, the proposed technique can enable non-expert users to
describe their problems in natural language, and the combination of LLMs and
SMT solvers can produce provably correct solutions.Comment: 25 pages, 7 figure
- …