109 research outputs found
Object-centric generative models for robot perception and action
The system of robot manipulation involves a pipeline consisting of the perception of objects in the environment and the planning of actions in 3D space. Deep learning approaches are employed to segment scenes into components of objects and then learn object-centric features to predict actions for downstream tasks. Despite having achieved promising performance in several manipulation tasks, supervised approaches lack inductive biases related to general properties of objects. Recent advances show that by encoding and reconstructing scenes in an object-centric fashion, the model can discover object-like entities from raw data without human supervision. Moreover, by reconstructing the discovered objects, the model can learn a variational latent space that captures the various shapes and textures of the objects, regularised by a chosen prior distribution. In this thesis, we investigate the properties of this learned object-centric latent space and develop novel object-centric generative models (OCGMs) that can be applied to real-world robotics scenarios.
In the first part of this thesis, we investigate a tool-synthesis task which leverages a learned latent space to optimise a wide range of tools applied to a reaching task. Given an image that illustrates the obstacles and the reaching target in the scene, an affordance predictor is trained to predict the feasibility of the tool for the given task. To imitate human tool-use experiences, feasibility labels are acquired from simulated trial-and-errors of the reaching task. We found that by employing an activation maximisation step, the model can synthesis proper tools for the given tasks with high accuracy. Moreover, the tool-synthesis process indicates the existence of a task-relevant trajectory in the learned latent space that can be found by a trained affordance predictor.
The second part of this thesis focuses on the development of novel OCGMs and their applications to robotic tasks. We first introduce a 2D OCGM that is deployed to robot manipulation datasets in both simulation and real-world scenarios. Despite the intensive interactions between robot arm and objects, we find the model discovers meaningful object entities from the raw observations without any human supervision. We next upgrade the 2D OCGM to 3D by leveraging NeRFs as decoders to explicitly model the 3D geometry of objects and the background. To disentangle the object spatial information from its appearance information, we propose a minimum volume principle for unsupervised 6D pose estimation of the objects. Considering the occlusion in the scene, we further improve the pose estimation by introducing a shape completion module to imagine the unobserved parts of the objects before the pose estimation step. In the end, we successfully apply the model in real-world robotics scenarios and compare its performance in several tasks including the 3D reconstruction, object-centric latent representation learning, 6D pose estimation for object rearrangement, against several baselines. We find that despite being an unsupervised approach, our model achieves improved performance across a range of different real-world tasks
An Empirical Study and Improvement for Speech Emotion Recognition
Multimodal speech emotion recognition aims to detect speakers' emotions from
audio and text. Prior works mainly focus on exploiting advanced networks to
model and fuse different modality information to facilitate performance, while
neglecting the effect of different fusion strategies on emotion recognition. In
this work, we consider a simple yet important problem: how to fuse audio and
text modality information is more helpful for this multimodal task. Further, we
propose a multimodal emotion recognition model improved by perspective loss.
Empirical results show our method obtained new state-of-the-art results on the
IEMOCAP dataset. The in-depth analysis explains why the improved model can
achieve improvements and outperforms baselines.Comment: Accepted by ICASSP 202
ObPose: Leveraging Pose for Object-Centric Scene Inference in 3D
We present ObPose, an unsupervised object-centric inference and generation
model which learns 3D-structured latent representations from RGB-D scenes.
Inspired by prior art in 2D representation learning, ObPose considers a
factorised latent space, separately encoding object location (where) and
appearance (what). ObPose further leverages an object's pose (i.e. location and
orientation), defined via a minimum volume principle, as a novel inductive bias
for learning the where component. To achieve this, we propose an efficient,
voxelised approximation approach to recover the object shape directly from a
neural radiance field (NeRF). As a consequence, ObPose models each scene as a
composition of NeRFs, richly representing individual objects. To evaluate the
quality of the learned representations, ObPose is evaluated quantitatively on
the YCB and CLEVR datatasets for unsupervised scene segmentation, outperforming
the current state-of-the-art in 3D scene inference (ObSuRF) by a significant
margin. Generative results provide qualitative demonstration that the same
ObPose model can both generate novel scenes and flexibly edit the objects in
them. These capacities again reflect the quality of the learned latents and the
benefits of disentangling the where and what components of a scene. Key design
choices made in the ObPose encoder are validated with ablations.Comment: 19 pages, 9 figure
The residential coal consumption : disparity in urban-rural China
We appreciate the support of the Program for Major Projects in Philosophy and Social Science Research of the Ministry of Education of China (No. 14JZD031), Key Program of National Social Science Fund of China (No. 15AJY005), National Natural Science Foundation of China (Nos. 71473203, 71171001, and 71471001), and New Century Excellent Talents in University (No. NCET-12-0595).Peer reviewedPostprin
Is the deformation rate of the Longmenshan fault zone really small? Insight from seismic data at the two-decade time scale
AxWin Transformer: A Context-Aware Vision Transformer Backbone with Axial Windows
Recently Transformer has shown good performance in several vision tasks due
to its powerful modeling capabilities. To reduce the quadratic complexity
caused by the attention, some outstanding work restricts attention to local
regions or extends axial interactions. However, these methos often lack the
interaction of local and global information, balancing coarse and fine-grained
information. To address this problem, we propose AxWin Attention, which models
context information in both local windows and axial views. Based on the AxWin
Attention, we develop a context-aware vision transformer backbone, named AxWin
Transformer, which outperforming the state-of-the-art methods in both
classification and downstream segmentation and detection tasks
TSST: A Benchmark and Evaluation Models for Text Speech-Style Transfer
Text style is highly abstract, as it encompasses various aspects of a
speaker's characteristics, habits, logical thinking, and the content they
express. However, previous text-style transfer tasks have primarily focused on
data-driven approaches, lacking in-depth analysis and research from the
perspectives of linguistics and cognitive science. In this paper, we introduce
a novel task called Text Speech-Style Transfer (TSST). The main objective is to
further explore topics related to human cognition, such as personality and
emotion, based on the capabilities of existing LLMs. Considering the objective
of our task and the distinctive characteristics of oral speech in real-life
scenarios, we trained multi-dimension (i.e. filler words, vividness,
interactivity, emotionality) evaluation models for the TSST and validated their
correlation with human assessments. We thoroughly analyze the performance of
several large language models (LLMs) and identify areas where further
improvement is needed. Moreover, driven by our evaluation models, we have
released a new corpus that improves the capabilities of LLMs in generating text
with speech-style characteristics. In summary, we present the TSST task, a new
benchmark for style transfer and emphasizing human-oriented evaluation,
exploring and advancing the performance of current LLMs.Comment: Working in progres
A Controllable Model of Grounded Response Generation
Current end-to-end neural conversation models inherently lack the flexibility
to impose semantic control in the response generation process, often resulting
in uninteresting responses. Attempts to boost informativeness alone come at the
expense of factual accuracy, as attested by pretrained language models'
propensity to "hallucinate" facts. While this may be mitigated by access to
background knowledge, there is scant guarantee of relevance and informativeness
in generated responses. We propose a framework that we call controllable
grounded response generation (CGRG), in which lexical control phrases are
either provided by a user or automatically extracted by a control phrase
predictor from dialogue context and grounding knowledge. Quantitative and
qualitative results show that, using this framework, a transformer based model
with a novel inductive attention mechanism, trained on a conversation-like
Reddit dataset, outperforms strong generation baselines.Comment: AAAI 202
- …