2,996 research outputs found
Language-Based Image Editing with Recurrent Attentive Models
We investigate the problem of Language-Based Image Editing (LBIE). Given a
source image and a natural language description, we want to generate a target
image by editing the source image based on the description. We propose a
generic modeling framework for two sub-tasks of LBIE: language-based image
segmentation and image colorization. The framework uses recurrent attentive
models to fuse image and language features. Instead of using a fixed step size,
we introduce for each region of the image a termination gate to dynamically
determine after each inference step whether to continue extrapolating
additional information from the textual description. The effectiveness of the
framework is validated on three datasets. First, we introduce a synthetic
dataset, called CoSaL, to evaluate the end-to-end performance of our LBIE
system. Second, we show that the framework leads to state-of-the-art
performance on image segmentation on the ReferIt dataset. Third, we present the
first language-based colorization result on the Oxford-102 Flowers dataset.Comment: Accepted to CVPR 2018 as a Spotligh
Audio-Infused Automatic Image Colorization by Exploiting Audio Scene Semantics
Automatic image colorization is inherently an ill-posed problem with
uncertainty, which requires an accurate semantic understanding of scenes to
estimate reasonable colors for grayscale images. Although recent
interaction-based methods have achieved impressive performance, it is still a
very difficult task to infer realistic and accurate colors for automatic
colorization. To reduce the difficulty of semantic understanding of grayscale
scenes, this paper tries to utilize corresponding audio, which naturally
contains extra semantic information about the same scene. Specifically, a novel
audio-infused automatic image colorization (AIAIC) network is proposed, which
consists of three stages. First, we take color image semantics as a bridge and
pretrain a colorization network guided by color image semantics. Second, the
natural co-occurrence of audio and video is utilized to learn the color
semantic correlations between audio and visual scenes. Third, the implicit
audio semantic representation is fed into the pretrained network to finally
realize the audio-guided colorization. The whole process is trained in a
self-supervised manner without human annotation. In addition, an audiovisual
colorization dataset is established for training and testing. Experiments
demonstrate that audio guidance can effectively improve the performance of
automatic colorization, especially for some scenes that are difficult to
understand only from visual modality
Example-based image colorization using locality consistent sparse representation
—Image colorization aims to produce a natural looking color image from a given grayscale image, which remains a challenging problem. In this paper, we propose a novel examplebased image colorization method exploiting a new locality consistent sparse representation. Given a single reference color image, our method automatically colorizes the target grayscale image by sparse pursuit. For efficiency and robustness, our method operates at the superpixel level. We extract low-level intensity features, mid-level texture features and high-level semantic features for each superpixel, which are then concatenated to form its descriptor. The collection of feature vectors for all the superpixels from the reference image composes the dictionary. We formulate colorization of target superpixels as a dictionary-based sparse reconstruction problem. Inspired by the observation that superpixels with similar spatial location and/or feature representation are likely to match spatially close regions from the reference image, we further introduce a locality promoting regularization term into the energy formulation which substantially improves the matching consistency and subsequent colorization results. Target superpixels are colorized based on the chrominance information from the dominant reference superpixels. Finally, to further improve coherence while preserving sharpness, we develop a new edge-preserving filter for chrominance channels with the guidance from the target grayscale image. To the best of our knowledge, this is the first work on sparse pursuit image colorization from single reference images. Experimental results demonstrate that our colorization method outperforms state-ofthe-art methods, both visually and quantitatively using a user stud
Estimation of Scribble Placement for Painting Colorization
Image colorization has been a topic of interest since
the mid 70’s and several algorithms have been proposed that
given a grayscale image and color scribbles (hints) produce a colorized image. Recently, this approach has been introduced in the field of art conservation and cultural heritage, where B&W photographs of paintings at previous stages have been colorized. However, the questions of what is the minimum number of scribbles necessary and where they should be placed in an image remain unexplored. Here we address this limitation using an iterative algorithm that provides insights as to the relationship between locally vs. globally important scribbles. Given a color image we randomly select scribbles and we attempt to color the
grayscale version of the original.We define a scribble contribution measure based on the reconstruction error. We demonstrate our approach using a widely used colorization algorithm and images from a Picasso painting and the peppers test image. We show that areas isolated by thick brushstrokes or areas with high textural variation are locally important but contribute very little to the
overall representation accuracy. We also find that for the case of Picasso on average 10% of scribble coverage is enough and that flat areas can be presented by few scribbles. The proposed method can be used verbatim to test any colorization algorithm
Pixelated Semantic Colorization
While many image colorization algorithms have recently shown the capability
of producing plausible color versions from gray-scale photographs, they still
suffer from limited semantic understanding. To address this shortcoming, we
propose to exploit pixelated object semantics to guide image colorization. The
rationale is that human beings perceive and distinguish colors based on the
semantic categories of objects. Starting from an autoregressive model, we
generate image color distributions, from which diverse colored results are
sampled. We propose two ways to incorporate object semantics into the
colorization model: through a pixelated semantic embedding and a pixelated
semantic generator. Specifically, the proposed convolutional neural network
includes two branches. One branch learns what the object is, while the other
branch learns the object colors. The network jointly optimizes a color
embedding loss, a semantic segmentation loss and a color generation loss, in an
end-to-end fashion. Experiments on PASCAL VOC2012 and COCO-stuff reveal that
our network, when trained with semantic segmentation labels, produces more
realistic and finer results compared to the colorization state-of-the-art
Probabilistic Image Colorization
We develop a probabilistic technique for colorizing grayscale natural images.
In light of the intrinsic uncertainty of this task, the proposed probabilistic
framework has numerous desirable properties. In particular, our model is able
to produce multiple plausible and vivid colorizations for a given grayscale
image and is one of the first colorization models to provide a proper
stochastic sampling scheme. Moreover, our training procedure is supported by a
rigorous theoretical framework that does not require any ad hoc heuristics and
allows for efficient modeling and learning of the joint pixel color
distribution. We demonstrate strong quantitative and qualitative experimental
results on the CIFAR-10 dataset and the challenging ILSVRC 2012 dataset
- …