105 research outputs found
Supporting Stylized Language Models Using Multi-Modality Features
As AI and machine learning systems become more common in our everyday lives, there is an increased desire to construct systems that are able to seamlessly interact and communicate with humans. This typically means creating systems that are able to communicate with humans via natural language. Given the variance of natural language, this can be a very challenging task. In this thesis, I explored the topic of humanlike language generation in the context of stylized language generation. Stylized language generation involves producing some text that exhibits a specific, desired style. In this dissertation, I specifically explored the use of multi-modality features as a means to provide sufficient information to produce high-quality stylized text output. I also explored how these multi-modality features can be used to identify and explain errors in the generated output. Finally, I constructed an automated language evaluation metric that can evaluate stylized language models
Referring Expression Comprehension: A Survey of Methods and Datasets
Referring expression comprehension (REC) aims to localize a target object in
an image described by a referring expression phrased in natural language.
Different from the object detection task that queried object labels have been
pre-defined, the REC problem only can observe the queries during the test. It
thus more challenging than a conventional computer vision problem. This task
has attracted a lot of attention from both computer vision and natural language
processing community, and several lines of work have been proposed, from
CNN-RNN model, modular network to complex graph-based model. In this survey, we
first examine the state of the art by comparing modern approaches to the
problem. We classify methods by their mechanism to encode the visual and
textual modalities. In particular, we examine the common approach of joint
embedding images and expressions to a common feature space. We also discuss
modular architectures and graph-based models that interface with structured
graph representation. In the second part of this survey, we review the datasets
available for training and evaluating REC systems. We then group results
according to the datasets, backbone models, settings so that they can be fairly
compared. Finally, we discuss promising future directions for the field, in
particular the compositional referring expression comprehension that requires
longer reasoning chain to address.Comment: Accepted to IEEE TM
Multi Sentence Description of Complex Manipulation Action Videos
Automatic video description requires the generation of natural language
statements about the actions, events, and objects in the video. An important
human trait, when we describe a video, is that we are able to do this with
variable levels of detail. Different from this, existing approaches for
automatic video descriptions are mostly focused on single sentence generation
at a fixed level of detail. Instead, here we address video description of
manipulation actions where different levels of detail are required for being
able to convey information about the hierarchical structure of these actions
relevant also for modern approaches of robot learning. We propose one hybrid
statistical and one end-to-end framework to address this problem. The hybrid
method needs much less data for training, because it models statistically
uncertainties within the video clips, while in the end-to-end method, which is
more data-heavy, we are directly connecting the visual encoder to the language
decoder without any intermediate (statistical) processing step. Both frameworks
use LSTM stacks to allow for different levels of description granularity and
videos can be described by simple single-sentences or complex multiple-sentence
descriptions. In addition, quantitative results demonstrate that these methods
produce more realistic descriptions than other competing approaches
- …