195 research outputs found
Joint bilateral learning for real-time universal photorealistic style transfer
Photorealistic style transfer is the task of transferring the
artistic style of an image onto a content target, producing a result that
is plausibly taken with a camera. Recent approaches, based on deep
neural networks, produce impressive results but are either too slow to
run at practical resolutions, or still contain objectionable artifacts. We
propose a new end-to-end model for photorealistic style transfer that is
both fast and inherently generates photorealistic results. The core of our
approach is a feed-forward neural network that learns local edge-aware
a ne transforms that automatically obey the photorealism constraint.
When trained on a diverse set of images and a variety of styles, our
model can robustly apply style transfer to an arbitrary pair of input
images. Compared to the state of the art, our method produces visually
superior results and is three orders of magnitude faster, enabling real-
time performance at 4K on a mobile phone. We validate our method
with ablation and user studies.Published versio
Deep Video Color Propagation
Traditional approaches for color propagation in videos rely on some form of
matching between consecutive video frames. Using appearance descriptors, colors
are then propagated both spatially and temporally. These methods, however, are
computationally expensive and do not take advantage of semantic information of
the scene. In this work we propose a deep learning framework for color
propagation that combines a local strategy, to propagate colors frame-by-frame
ensuring temporal stability, and a global strategy, using semantics for color
propagation within a longer range. Our evaluation shows the superiority of our
strategy over existing video and image color propagation methods as well as
neural photo-realistic style transfer approaches.Comment: BMVC 201
Instant Photorealistic Style Transfer: A Lightweight and Adaptive Approach
In this paper, we propose an Instant Photorealistic Style Transfer (IPST)
approach, designed to achieve instant photorealistic style transfer on
super-resolution inputs without the need for pre-training on pair-wise datasets
or imposing extra constraints. Our method utilizes a lightweight StyleNet to
enable style transfer from a style image to a content image while preserving
non-color information. To further enhance the style transfer process, we
introduce an instance-adaptive optimization to prioritize the photorealism of
outputs and accelerate the convergence of the style network, leading to a rapid
training completion within seconds. Moreover, IPST is well-suited for
multi-frame style transfer tasks, as it retains temporal and multi-view
consistency of the multi-frame inputs such as video and Neural Radiance Field
(NeRF). Experimental results demonstrate that IPST requires less GPU memory
usage, offers faster multi-frame transfer speed, and generates photorealistic
outputs, making it a promising solution for various photorealistic transfer
applications.Comment: 8 pages (reference excluded), 6 figures, 4 table
Deep representation learning for photorealistic content creation
We study the problem of deep representation learning for photorealistic content creation. This is a critical component in many computer vision applications ranging from virtual reality, videography, and even retail and advertising. In this thesis, we use deep neural techniques to develop end-to-end models that are capable of generating photorealistic results. Our framework is applied in three applications.
First, we study real-time universal Photorealistic Image Style Transfer. Photorealistic style transfer is the task of transferring the artistic style of an image onto a content target, producing a result that is plausibly taken with a camera. We propose a new end-to-end model for photorealistic style transfer that is both fast and inherently generates photorealistic results. The core of our approach is a feed-forward neural network that learns local edge-aware affine transforms that automatically obey the photorealism constraint. Our method produces visually superior results and is three orders of magnitude faster, enabling real-time performance at 4K on a mobile phone.
Next, we learn real-time localized Photorealistic Video Style Transfer. We present a novel algorithm for transferring artistic styles of an image onto local regions of a target video while preserving its photorealism. Local regions may be selected either fully automatically from an image, through using video segmentation algorithms, or from casual user guidance such as scribbles. Our method is real-time and works on arbitrary inputs without runtime optimization once trained. We demonstrate our method on a variety of style images and target videos, including the ability to transfer different styles onto multiple objects simultaneously, and smoothly transition between styles in time.
Lastly, we tackle the problem of attribute-based Fashion Image Retrieval and Content Creation. We present an effective approach for generating new outfits based on the input queries through generative adversarial learning. We address this challenge by decomposing the complicated process into two stages. In the first stage, we present a novel attribute-aware global ranking network for attribute-based fashion retrieval. In the second stage, a generative model is used to finalize the retrieved results conditioned on an individual’s preferred style. We demonstrate promising results on standard large-scale benchmarks
Advancements and Trends in Ultra-High-Resolution Image Processing: An Overview
Currently, to further improve visual enjoyment, Ultra-High-Definition (UHD)
images are catching wide attention. Here, UHD images are usually referred to as
having a resolution greater than or equal to . However, since
the imaging equipment is subject to environmental noise or equipment jitter,
UHD images are prone to contrast degradation, blurring, low dynamic range, etc.
To address these issues, a large number of algorithms for UHD image enhancement
have been proposed. In this paper, we introduce the current state of UHD image
enhancement from two perspectives, one is the application field and the other
is the technology. In addition, we briefly explore its trends
Neural Preset for Color Style Transfer
In this paper, we present a Neural Preset technique to address the
limitations of existing color style transfer methods, including visual
artifacts, vast memory requirement, and slow style switching speed. Our method
is based on two core designs. First, we propose Deterministic Neural Color
Mapping (DNCM) to consistently operate on each pixel via an image-adaptive
color mapping matrix, avoiding artifacts and supporting high-resolution inputs
with a small memory footprint. Second, we develop a two-stage pipeline by
dividing the task into color normalization and stylization, which allows
efficient style switching by extracting color styles as presets and reusing
them on normalized input images. Due to the unavailability of pairwise
datasets, we describe how to train Neural Preset via a self-supervised
strategy. Various advantages of Neural Preset over existing methods are
demonstrated through comprehensive evaluations. Notably, Neural Preset enables
stable 4K color style transfer in real-time without artifacts. Besides, we show
that our trained model can naturally support multiple applications without
fine-tuning, including low-light image enhancement, underwater image
correction, image dehazing, and image harmonization. Project page with demos:
https://zhkkke.github.io/NeuralPreset .Comment: Project page with demos: https://zhkkke.github.io/NeuralPreset .
Artifact-free real-time 4K color style transfer via AI-generated presets.
CVPR 202
Modernizing Old Photos Using Multiple References via Photorealistic Style Transfer
This paper firstly presents old photo modernization using multiple references
by performing stylization and enhancement in a unified manner. In order to
modernize old photos, we propose a novel multi-reference-based old photo
modernization (MROPM) framework consisting of a network MROPM-Net and a novel
synthetic data generation scheme. MROPM-Net stylizes old photos using multiple
references via photorealistic style transfer (PST) and further enhances the
results to produce modern-looking images. Meanwhile, the synthetic data
generation scheme trains the network to effectively utilize multiple references
to perform modernization. To evaluate the performance, we propose a new old
photos benchmark dataset (CHD) consisting of diverse natural indoor and outdoor
scenes. Extensive experiments show that the proposed method outperforms other
baselines in performing modernization on real old photos, even though no old
photos were used during training. Moreover, our method can appropriately select
styles from multiple references for each semantic region in the old photo to
further improve the modernization performance.Comment: Accepted to CVPR 2023. Website:
https://kaist-viclab.github.io/old-photo-modernizatio
- …