29 research outputs found
Content-Adaptive Sketch Portrait Generation by Decompositional Representation Learning
Sketch portrait generation benefits a wide range of applications such as
digital entertainment and law enforcement. Although plenty of efforts have been
dedicated to this task, several issues still remain unsolved for generating
vivid and detail-preserving personal sketch portraits. For example, quite a few
artifacts may exist in synthesizing hairpins and glasses, and textural details
may be lost in the regions of hair or mustache. Moreover, the generalization
ability of current systems is somewhat limited since they usually require
elaborately collecting a dictionary of examples or carefully tuning
features/components. In this paper, we present a novel representation learning
framework that generates an end-to-end photo-sketch mapping through structure
and texture decomposition. In the training stage, we first decompose the input
face photo into different components according to their representational
contents (i.e., structural and textural parts) by using a pre-trained
Convolutional Neural Network (CNN). Then, we utilize a Branched Fully
Convolutional Neural Network (BFCN) for learning structural and textural
representations, respectively. In addition, we design a Sorted Matching Mean
Square Error (SM-MSE) metric to measure texture patterns in the loss function.
In the stage of sketch rendering, our approach automatically generates
structural and textural representations for the input photo and produces the
final result via a probabilistic fusion scheme. Extensive experiments on
several challenging benchmarks suggest that our approach outperforms
example-based synthesis algorithms in terms of both perceptual and objective
metrics. In addition, the proposed method also has better generalization
ability across dataset without additional training.Comment: Published in TIP 201
End-to-End Photo-Sketch Generation via Fully Convolutional Representation Learning
Sketch-based face recognition is an interesting task in vision and multimedia
research, yet it is quite challenging due to the great difference between face
photos and sketches. In this paper, we propose a novel approach for
photo-sketch generation, aiming to automatically transform face photos into
detail-preserving personal sketches. Unlike the traditional models synthesizing
sketches based on a dictionary of exemplars, we develop a fully convolutional
network to learn the end-to-end photo-sketch mapping. Our approach takes whole
face photos as inputs and directly generates the corresponding sketch images
with efficient inference and learning, in which the architecture are stacked by
only convolutional kernels of very small sizes. To well capture the person
identity during the photo-sketch transformation, we define our optimization
objective in the form of joint generative-discriminative minimization. In
particular, a discriminative regularization term is incorporated into the
photo-sketch generation, enhancing the discriminability of the generated person
sketches against other individuals. Extensive experiments on several standard
benchmarks suggest that our approach outperforms other state-of-the-art methods
in both photo-sketch generation and face sketch verification.Comment: 8 pages, 6 figures. Proceeding in ACM International Conference on
Multimedia Retrieval (ICMR), 201
Semi-supervised Cycle-GAN for face photo-sketch translation in the wild
The performance of face photo-sketch translation has improved a lot thanks to
deep neural networks. GAN based methods trained on paired images can produce
high-quality results under laboratory settings. Such paired datasets are,
however, often very small and lack diversity. Meanwhile, Cycle-GANs trained
with unpaired photo-sketch datasets suffer from the \emph{steganography}
phenomenon, which makes them not effective to face photos in the wild. In this
paper, we introduce a semi-supervised approach with a noise-injection strategy,
named Semi-Cycle-GAN (SCG), to tackle these problems. For the first problem, we
propose a {\em pseudo sketch feature} representation for each input photo
composed from a small reference set of photo-sketch pairs, and use the
resulting {\em pseudo pairs} to supervise a photo-to-sketch generator
. The outputs of can in turn help to train a sketch-to-photo
generator in a self-supervised manner. This allows us to train
and using a small reference set of photo-sketch pairs
together with a large face photo dataset (without ground-truth sketches). For
the second problem, we show that the simple noise-injection strategy works well
to alleviate the \emph{steganography} effect in SCG and helps to produce more
reasonable sketch-to-photo results with less overfitting than fully supervised
approaches. Experiments show that SCG achieves competitive performance on
public benchmarks and superior results on photos in the wild.Comment: 11 pages, 11 figures, 5 tables (+ 7 page appendix
Forensic face photo-sketch recognition using a deep learning-based architecture
Numerous methods that automatically identify subjects depicted in sketches as described by eyewitnesses have been implemented, but their performance often degrades when using real-world forensic sketches and extended galleries that mimic law enforcement mug-shot galleries. Moreover, little work has been done to apply deep learning for face photo-sketch recognition despite its success in numerous application domains including traditional face recognition. This is primarily due to the limited number of sketch images available, which are insufficient to robustly train large networks. This letter aims to tackle these issues with the following contributions: 1) a state-of-the-art model pre-trained for face photo recognition is tuned for face photo-sketch recognition by applying transfer learning, 2) a three-dimensional morphable model is used to synthesise new images and artificially expand the training data, allowing the network to prevent over-fitting and learn better features, 3) multiple synthetic sketches are also used in the testing stage to improve performance, and 4) fusion of the proposed method with a state-of-the-art algorithm is shown to further boost performance. An extensive evaluation of several popular and state-of-the-art algorithms is also performed using publicly available datasets, thereby serving as a benchmark for future algorithms. Compared to a leading method, the proposed framework is shown to reduce the error rate by 80.7% for viewed sketches and lowers the mean retrieval rank by 32.5% for real-world forensic sketches.peer-reviewe