5,803 research outputs found
Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis
This paper studies a combination of generative Markov random field (MRF)
models and discriminatively trained deep convolutional neural networks (dCNNs)
for synthesizing 2D images. The generative MRF acts on higher-levels of a dCNN
feature pyramid, controling the image layout at an abstract level. We apply the
method to both photographic and non-photo-realistic (artwork) synthesis tasks.
The MRF regularizer prevents over-excitation artifacts and reduces implausible
feature mixtures common to previous dCNN inversion approaches, permitting
synthezing photographic content with increased visual plausibility. Unlike
standard MRF-based texture synthesis, the combined system can both match and
adapt local features with considerable variability, yielding results far out of
reach of classic generative MRF methods.Comment: 9 pages, 9 figure
SurReal: enhancing Surgical simulation Realism using style transfer
Surgical simulation is an increasingly important element of surgical
education. Using simulation can be a means to address some of the significant
challenges in developing surgical skills with limited time and resources. The
photo-realistic fidelity of simulations is a key feature that can improve the
experience and transfer ratio of trainees. In this paper, we demonstrate how we
can enhance the visual fidelity of existing surgical simulation by performing
style transfer of multi-class labels from real surgical video onto synthetic
content. We demonstrate our approach on simulations of cataract surgery using
real data labels from an existing public dataset. Our results highlight the
feasibility of the approach and also the powerful possibility to extend this
technique to incorporate additional temporal constraints and to different
applications
Controllable Artistic Text Style Transfer via Shape-Matching GAN
Artistic text style transfer is the task of migrating the style from a source
image to the target text to create artistic typography. Recent style transfer
methods have considered texture control to enhance usability. However,
controlling the stylistic degree in terms of shape deformation remains an
important open challenge. In this paper, we present the first text style
transfer network that allows for real-time control of the crucial stylistic
degree of the glyph through an adjustable parameter. Our key contribution is a
novel bidirectional shape matching framework to establish an effective
glyph-style mapping at various deformation levels without paired ground truth.
Based on this idea, we propose a scale-controllable module to empower a single
network to continuously characterize the multi-scale shape features of the
style image and transfer these features to the target text. The proposed method
demonstrates its superiority over previous state-of-the-arts in generating
diverse, controllable and high-quality stylized text.Comment: Accepted by ICCV 2019. Code is available at
https://williamyang1991.github.io/projects/ICCV2019/SMGAN.htm
Painterly rendering techniques: A state-of-the-art review of current approaches
In this publication we will look at the different methods presented over the past few decades which attempt to recreate digital paintings. While previous surveys concentrate on the broader subject of non-photorealistic rendering, the focus of this paper is firmly placed on painterly rendering techniques. We compare different methods used to produce different output painting styles such as abstract, colour pencil, watercolour, oriental, oil and pastel. Whereas some methods demand a high level of interaction using a skilled artist, others require simple parameters provided by a user with little or no artistic experience. Many methods attempt to provide more automation with the use of varying forms of reference data. This reference data can range from still photographs, video, 3D polygonal meshes or even 3D point clouds. The techniques presented here endeavour to provide tools and styles that are not traditionally available to an artist. Copyright © 2012 John Wiley & Sons, Ltd
Generative Image Inpainting with Contextual Attention
Recent deep learning based approaches have shown promising results for the
challenging task of inpainting large missing regions in an image. These methods
can generate visually plausible image structures and textures, but often create
distorted structures or blurry textures inconsistent with surrounding areas.
This is mainly due to ineffectiveness of convolutional neural networks in
explicitly borrowing or copying information from distant spatial locations. On
the other hand, traditional texture and patch synthesis approaches are
particularly suitable when it needs to borrow textures from the surrounding
regions. Motivated by these observations, we propose a new deep generative
model-based approach which can not only synthesize novel image structures but
also explicitly utilize surrounding image features as references during network
training to make better predictions. The model is a feed-forward, fully
convolutional neural network which can process images with multiple holes at
arbitrary locations and with variable sizes during the test time. Experiments
on multiple datasets including faces (CelebA, CelebA-HQ), textures (DTD) and
natural images (ImageNet, Places2) demonstrate that our proposed approach
generates higher-quality inpainting results than existing ones. Code, demo and
models are available at: https://github.com/JiahuiYu/generative_inpainting.Comment: Accepted in CVPR 2018; add CelebA-HQ results; open sourced;
interactive demo available: http://jhyu.me/dem
LADN: Local Adversarial Disentangling Network for Facial Makeup and De-Makeup
We propose a local adversarial disentangling network (LADN) for facial makeup
and de-makeup. Central to our method are multiple and overlapping local
adversarial discriminators in a content-style disentangling network for
achieving local detail transfer between facial images, with the use of
asymmetric loss functions for dramatic makeup styles with high-frequency
details. Existing techniques do not demonstrate or fail to transfer
high-frequency details in a global adversarial setting, or train a single local
discriminator only to ensure image structure consistency and thus work only for
relatively simple styles. Unlike others, our proposed local adversarial
discriminators can distinguish whether the generated local image details are
consistent with the corresponding regions in the given reference image in
cross-image style transfer in an unsupervised setting. Incorporating these
technical contributions, we achieve not only state-of-the-art results on
conventional styles but also novel results involving complex and dramatic
styles with high-frequency details covering large areas across multiple facial
features. A carefully designed dataset of unpaired before and after makeup
images is released.Comment: Qiao and Guanzhi have equal contribution. Accepted to ICCV 2019.
Project website: https://georgegu1997.github.io/LADN-project-page
TileGAN: Synthesis of Large-Scale Non-Homogeneous Textures
We tackle the problem of texture synthesis in the setting where many input
images are given and a large-scale output is required. We build on recent
generative adversarial networks and propose two extensions in this paper.
First, we propose an algorithm to combine outputs of GANs trained on a smaller
resolution to produce a large-scale plausible texture map with virtually no
boundary artifacts. Second, we propose a user interface to enable artistic
control. Our quantitative and qualitative results showcase the generation of
synthesized high-resolution maps consisting of up to hundreds of megapixels as
a case in point.Comment: Code is available at http://github.com/afruehstueck/tileGA
Cross-Platform Presentation of Interactive Volumetric Imagery
Volume data is useful across many disciplines, not just medicine.
Thus, it is very important that researchers have a simple and
lightweight method of sharing and reproducing such volumetric
data. In this paper, we explore some of the challenges associated
with volume rendering, both from a classical sense and from the
context of Web3D technologies. We describe and evaluate the pro-
posed X3D Volume Rendering Component and its associated styles
for their suitability in the visualization of several types of image
data. Additionally, we examine the ability for a minimal X3D node
set to capture provenance and semantic information from outside
ontologies in metadata and integrate it with the scene graph
Neural Style Transfer: A Review
The seminal work of Gatys et al. demonstrated the power of Convolutional
Neural Networks (CNNs) in creating artistic imagery by separating and
recombining image content and style. This process of using CNNs to render a
content image in different styles is referred to as Neural Style Transfer
(NST). Since then, NST has become a trending topic both in academic literature
and industrial applications. It is receiving increasing attention and a variety
of approaches are proposed to either improve or extend the original NST
algorithm. In this paper, we aim to provide a comprehensive overview of the
current progress towards NST. We first propose a taxonomy of current algorithms
in the field of NST. Then, we present several evaluation methods and compare
different NST algorithms both qualitatively and quantitatively. The review
concludes with a discussion of various applications of NST and open problems
for future research. A list of papers discussed in this review, corresponding
codes, pre-trained models and more comparison results are publicly available at
https://github.com/ycjing/Neural-Style-Transfer-Papers.Comment: Project page: https://github.com/ycjing/Neural-Style-Transfer-Paper
Hierarchy Composition GAN for High-fidelity Image Synthesis
Despite the rapid progress of generative adversarial networks (GANs) in image
synthesis in recent years, the existing image synthesis approaches work in
either geometry domain or appearance domain alone which often introduces
various synthesis artifacts. This paper presents an innovative Hierarchical
Composition GAN (HIC-GAN) that incorporates image synthesis in geometry and
appearance domains into an end-to-end trainable network and achieves superior
synthesis realism in both domains simultaneously. We design an innovative
hierarchical composition mechanism that is capable of learning realistic
composition geometry and handling occlusions while multiple foreground objects
are involved in image composition. In addition, we introduce a novel attention
mask mechanism that guides to adapt the appearance of foreground objects which
also helps to provide better training reference for learning in geometry
domain. Extensive experiments on scene text image synthesis, portrait editing
and indoor rendering tasks show that the proposed HIC-GAN achieves superior
synthesis performance qualitatively and quantitatively.Comment: 11 pages, 8 figure
- …