18 research outputs found
Learning a natural-language to LTL executable semantic parser for grounded robotics
Children acquire their native language with apparent ease by observing how
language is used in context and attempting to use it themselves. They do so
without laborious annotations, negative examples, or even direct corrections.
We take a step toward robots that can do the same by training a grounded
semantic parser, which discovers latent linguistic representations that can be
used for the execution of natural-language commands. In particular, we focus on
the difficult domain of commands with a temporal aspect, whose semantics we
capture with Linear Temporal Logic, LTL. Our parser is trained with pairs of
sentences and executions as well as an executor. At training time, the parser
hypothesizes a meaning representation for the input as a formula in LTL. Three
competing pressures allow the parser to discover meaning from language. First,
any hypothesized meaning for a sentence must be permissive enough to reflect
all the annotated execution trajectories. Second, the executor -- a pretrained
end-to-end LTL planner -- must find that the observe trajectories are likely
executions of the meaning. Finally, a generator, which reconstructs the
original input, encourages the model to find representations that conserve
knowledge about the command. Together these ensure that the meaning is neither
too general nor too specific. Our model generalizes well, being able to parse
and execute both machine-generated and human-generated commands, with
near-equal accuracy, despite the fact that the human-generated sentences are
much more varied and complex with an open lexicon. The approach presented here
is not specific to LTL: it can be applied to any domain where sentence meanings
can be hypothesized and an executor can verify these meanings, thus opening the
door to many applications for robotic agents.Comment: 10 pages, 2 figures, Accepted in Conference on Robot Learning (CoRL)
202
A Type-coherent, Expressive Representation as an Initial Step to Language Understanding
A growing interest in tasks involving language understanding by the NLP
community has led to the need for effective semantic parsing and inference.
Modern NLP systems use semantic representations that do not quite fulfill the
nuanced needs for language understanding: adequately modeling language
semantics, enabling general inferences, and being accurately recoverable. This
document describes underspecified logical forms (ULF) for Episodic Logic (EL),
which is an initial form for a semantic representation that balances these
needs. ULFs fully resolve the semantic type structure while leaving issues such
as quantifier scope, word sense, and anaphora unresolved; they provide a
starting point for further resolution into EL, and enable certain structural
inferences without further resolution. This document also presents preliminary
results of creating a hand-annotated corpus of ULFs for the purpose of training
a precise ULF parser, showing a three-person pairwise interannotator agreement
of 0.88 on confident annotations. We hypothesize that a divide-and-conquer
approach to semantic parsing starting with derivation of ULFs will lead to
semantic analyses that do justice to subtle aspects of linguistic meaning, and
will enable construction of more accurate semantic parsers.Comment: Accepted for publication at The 13th International Conference on
Computational Semantics (IWCS 2019
Deep compositional robotic planners that follow natural language commands
We demonstrate how a sampling-based robotic planner can be augmented to learn
to understand a sequence of natural language commands in a continuous
configuration space to move and manipulate objects. Our approach combines a
deep network structured according to the parse of a complex command that
includes objects, verbs, spatial relations, and attributes, with a
sampling-based planner, RRT. A recurrent hierarchical deep network controls how
the planner explores the environment, determines when a planned path is likely
to achieve a goal, and estimates the confidence of each move to trade off
exploitation and exploration between the network and the planner. Planners are
designed to have near-optimal behavior when information about the task is
missing, while networks learn to exploit observations which are available from
the environment, making the two naturally complementary. Combining the two
enables generalization to new maps, new kinds of obstacles, and more complex
sentences that do not occur in the training set. Little data is required to
train the model despite it jointly acquiring a CNN that extracts features from
the environment as it learns the meanings of words. The model provides a level
of interpretability through the use of attention maps allowing users to see its
reasoning steps despite being an end-to-end model. This end-to-end model allows
robots to learn to follow natural language commands in challenging continuous
environments.Comment: Accepted in ICRA 202
Knowledge and Reasoning for Image Understanding
abstract: Image Understanding is a long-established discipline in computer vision, which encompasses a body of advanced image processing techniques, that are used to locate (“where”), characterize and recognize (“what”) objects, regions, and their attributes in the image. However, the notion of “understanding” (and the goal of artificial intelligent machines) goes beyond factual recall of the recognized components and includes reasoning and thinking beyond what can be seen (or perceived). Understanding is often evaluated by asking questions of increasing difficulty. Thus, the expected functionalities of an intelligent Image Understanding system can be expressed in terms of the functionalities that are required to answer questions about an image. Answering questions about images require primarily three components: Image Understanding, question (natural language) understanding, and reasoning based on knowledge. Any question, asking beyond what can be directly seen, requires modeling of commonsense (or background/ontological/factual) knowledge and reasoning.
Knowledge and reasoning have seen scarce use in image understanding applications. In this thesis, we demonstrate the utilities of incorporating background knowledge and using explicit reasoning in image understanding applications. We first present a comprehensive survey of the previous work that utilized background knowledge and reasoning in understanding images. This survey outlines the limited use of commonsense knowledge in high-level applications. We then present a set of vision and reasoning-based methods to solve several applications and show that these approaches benefit in terms of accuracy and interpretability from the explicit use of knowledge and reasoning. We propose novel knowledge representations of image, knowledge acquisition methods, and a new implementation of an efficient probabilistic logical reasoning engine that can utilize publicly available commonsense knowledge to solve applications such as visual question answering, image puzzles. Additionally, we identify the need for new datasets that explicitly require external commonsense knowledge to solve. We propose the new task of Image Riddles, which requires a combination of vision, and reasoning based on ontological knowledge; and we collect a sufficiently large dataset to serve as an ideal testbed for vision and reasoning research. Lastly, we propose end-to-end deep architectures that can combine vision, knowledge and reasoning modules together and achieve large performance boosts over state-of-the-art methods.Dissertation/ThesisDoctoral Dissertation Computer Science 201