61 research outputs found
MOGAN: Morphologic-structure-aware Generative Learning from a Single Image
In most interactive image generation tasks, given regions of interest (ROI)
by users, the generated results are expected to have adequate diversities in
appearance while maintaining correct and reasonable structures in original
images. Such tasks become more challenging if only limited data is available.
Recently proposed generative models complete training based on only one image.
They pay much attention to the monolithic feature of the sample while ignoring
the actual semantic information of different objects inside the sample. As a
result, for ROI-based generation tasks, they may produce inappropriate samples
with excessive randomicity and without maintaining the related objects' correct
structures. To address this issue, this work introduces a
MOrphologic-structure-aware Generative Adversarial Network named MOGAN that
produces random samples with diverse appearances and reliable structures based
on only one image. For training for ROI, we propose to utilize the data coming
from the original image being augmented and bring in a novel module to
transform such augmented data into knowledge containing both structures and
appearances, thus enhancing the model's comprehension of the sample. To learn
the rest areas other than ROI, we employ binary masks to ensure the generation
isolated from ROI. Finally, we set parallel and hierarchical branches of the
mentioned learning process. Compared with other single image GAN schemes, our
approach focuses on internal features including the maintenance of rational
structures and variation on appearance. Experiments confirm a better capacity
of our model on ROI-based image generation tasks than its competitive peers
Exemplar-Based Texture Synthesis Using Two Random Coefficients Autoregressive Models
Example-based texture synthesis is a fundamental topic of many image analysis and computer vision applications. Consequently, its representation is one of the most critical and challenging topics in computer vision and pattern recognition, attracting much academic interest throughout the years. In this paper, a new statistical method to synthesize textures is proposed. It consists in using two indexed random coefficients autoregressive (2D-RCA) models to deal with this problem. These models have a good ability to well detect neighborhood information. Simulations have demonstrated that the 2D-RCA models are very suitable to represent textures. So, in this work, to generate textures from an example, each original image is splitted into blocks which are modeled by the 2D-RCA. The proposed algorithm produces approximations of the obtained blocks images from the original image using the generalized method of moments (GMM). Different sizes of windows have been used. This study offers some important insights into the newly generated image. Satisfying obtained results have been compared to those given by well-established methods. The proposed algorithm outperforms the state-of-the-art approaches
Generative Model with Coordinate Metric Learning for Object Recognition Based on 3D Models
Given large amount of real photos for training, Convolutional neural network
shows excellent performance on object recognition tasks. However, the process
of collecting data is so tedious and the background are also limited which
makes it hard to establish a perfect database. In this paper, our generative
model trained with synthetic images rendered from 3D models reduces the
workload of data collection and limitation of conditions. Our structure is
composed of two sub-networks: semantic foreground object reconstruction network
based on Bayesian inference and classification network based on multi-triplet
cost function for avoiding over-fitting problem on monotone surface and fully
utilizing pose information by establishing sphere-like distribution of
descriptors in each category which is helpful for recognition on regular photos
according to poses, lighting condition, background and category information of
rendered images. Firstly, our conjugate structure called generative model with
metric learning utilizing additional foreground object channels generated from
Bayesian rendering as the joint of two sub-networks. Multi-triplet cost
function based on poses for object recognition are used for metric learning
which makes it possible training a category classifier purely based on
synthetic data. Secondly, we design a coordinate training strategy with the
help of adaptive noises acting as corruption on input images to help both
sub-networks benefit from each other and avoid inharmonious parameter tuning
due to different convergence speed of two sub-networks. Our structure achieves
the state of the art accuracy of over 50\% on ShapeNet database with data
migration obstacle from synthetic images to real photos. This pipeline makes it
applicable to do recognition on real images only based on 3D models.Comment: 14 page
Towards Interaction-level Video Action Understanding
A huge amount of videos have been created, spread, and viewed daily. Among these massive videos, the actions and activities of humans account for a large part. We desire machines to understand human actions in videos as this is essential to various applications, including but not limited to autonomous driving cars, security systems, human-robot interactions and healthcare. Towards real intelligent system that is able to interact with humans, video understanding must go beyond simply answering ``what is the action in the video", but be more aware of what those actions mean to humans and be more in line with human thinking, which we call interactive-level action understanding. This thesis identifies three main challenges to approaching interactive-level video action understanding: 1) understanding actions given human consensus; 2) understanding actions based on specific human rules; 3) directly understanding actions in videos via human natural language. For the first challenge, we select video summary as a representative task that aims to select informative frames to retain high-level information based on human annotators' experience. Through self-attention architecture and meta-learning, which jointly process dual representations of visual and sequential information for video summarization, the proposed model is capable of understanding video from human consensus (e.g., how humans think which parts of an action sequence are essential). For the second challenge, our works on action quality assessment utilize transformer decoders to parse the input action into several sub-actions and assess the more fine-grained qualities of the given action, yielding the capability of action understanding given specific human rules. (e.g., how well a diving action performs, how well a robot performs surgery) The third key idea explored in this thesis is to use graph neural networks in an adversarial fashion to understand actions through natural language. We demonstrate the utility of this technique for the video captioning task, which takes an action video as input, outputs natural language, and yields state-of-the-art performance. It can be concluded that the research directions and methods introduced in this thesis provide fundamental components toward interactive-level action understanding
- …