Recovering Faces from Portraits with Auxiliary Facial Attributes

Recovering a photorealistic face from an artistic portrait is a challenging task since crucial facial details are often distorted or completely lost in artistic compositions. To handle this loss, we propose an Attribute-guided Face Recovery from Portraits (AFRP) that utilizes a Face Recovery Network (FRN) and a Discriminative Network (DN). FRN consists of an autoencoder with residual block-embedded skip-connections and incorporates facial attribute vectors into the feature maps of input portraits at the bottleneck of the autoencoder. DN has multiple convolutional and fully-connected layers, and its role is to enforce FRN to generate authentic face images with corresponding facial attributes dictated by the input attribute vectors. %Leveraging on the spatial transformer networks, FRN automatically compensates for misalignments of portraits. % and generates aligned face images. For the preservation of identities, we impose the recovered and ground-truth faces to share similar visual features. Specifically, DN determines whether the recovered image looks like a real face and checks if the facial attributes extracted from the recovered image are consistent with given attributes. %Our method can recover high-quality photorealistic faces from unaligned portraits while preserving the identity of the face images as well as it can reconstruct a photorealistic face image with a desired set of attributes. Our method can recover photorealistic identity-preserving faces with desired attributes from unseen stylized portraits, artistic paintings, and hand-drawn sketches. On large-scale synthesized and sketch datasets, we demonstrate that our face recovery method achieves state-of-the-art results.Comment: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV

Face Hallucination via Deep Neural Networks.

We firstly address aligned low-resolution (LR) face images (i.e. 16X16 pixels) by designing a discriminative generative network, named URDGN. URDGN is composed of two networks: a generative model and a discriminative model. We introduce a pixel-wise L2 regularization term to the generative model and exploit the feedback of the discriminative network to make the upsampled face images more similar to real ones. We present an end-to-end transformative discriminative neural network (TDN) devised for super-resolving unaligned tiny face images. TDN embeds spatial transformation layers to enforce local receptive fields to line-up with similar spatial supports. To upsample noisy unaligned LR face images, we propose decoder-encoder-decoder networks. A transformative discriminative decoder network is employed to upsample and denoise LR inputs simultaneously. Then we project the intermediate HR faces to aligned and noise-free LR faces by a transformative encoder network. Finally, high-quality hallucinated HR images are generated by our second decoder. Furthermore, we present an end-to-end multiscale transformative discriminative neural network (MTDN) to super-resolve unaligned LR face images of different resolutions in a unified framework. We propose a method that explicitly incorporates structural information of faces into the face super-resolution process by using a multi-task convolutional neural network (CNN). Our method not only uses low-level information (i.e. intensity similarity), but also middle-level information (i.e. face structure) to further explore spatial constraints of facial components from LR inputs images. We demonstrate that supplementing residual images or feature maps with additional facial attribute information can significantly reduce the ambiguity in face super-resolution. To explore this idea, we develop an attribute-embedded upsampling network. In this manner, our method is able to super-resolve LR faces by a large upscaling factor while reducing the uncertainty of one-to-many mappings remarkably. We further push the boundaries of hallucinating a tiny, non-frontal face image to understand how much of this is possible by leveraging the availability of large datasets and deep networks. To this end, we introduce a novel Transformative Adversarial Neural Network (TANN) to jointly frontalize very LR out-of-plane rotated face images (including profile views) and aggressively super-resolve them by 8X, regardless of their original poses and without using any 3D information. Besides recovering an HR face images from an LR version, this thesis also addresses the task of restoring realistic faces from stylized portrait images, which can also be regarded as face hallucination

Identity-preserving Face Recovery from Portraits

Recovering the latent photorealistic faces from their artistic portraits aids human perception and facial analysis. However, a recovery process that can preserve identity is challenging because the fine details of real faces can be distorted or lost in stylized images. In this paper, we present a new Identity-preserving Face Recovery from Portraits (IFRP) to recover latent photorealistic faces from unaligned stylized portraits. Our IFRP method consists of two components: Style Removal Network (SRN) and Discriminative Network (DN). The SRN is designed to transfer feature maps of stylized images to the feature maps of the corresponding photorealistic faces. By embedding spatial transformer networks into the SRN, our method can compensate for misalignments of stylized faces automatically and output aligned realistic face images. The role of the DN is to enforce recovered faces to be similar to authentic faces. To ensure the identity preservation, we promote the recovered and ground-truth faces to share similar visual features via a distance measure which compares features of recovered and ground-truth faces extracted from a pre-trained VGG network. We evaluate our method on a large-scale synthesized dataset of real and stylized face pairs and attain state of the art results. In addition, our method can recover photorealistic faces from previously unseen stylized portraits, original paintings and human-drawn sketches

Face Destylization

Numerous style transfer methods which produce artistic styles of portraits have been proposed to date. However, the inverse problem of converting the stylized portraits back into realistic faces is yet to be investigated thoroughly. Reverting an artistic portrait to its original photo-realistic face image has potential to facilitate human perception and identity analysis. In this paper, we propose a novel Face Destylization Neural Network (FDNN) to restore the latent photo-realistic faces from the stylized ones. We develop a Style Removal Network composed of convolutional, fully-connected and deconvolutional layers. The convolutional layers are designed to extract facial components from stylized face images. Consecutively, the fully-connected layer transfers the extracted feature maps of stylized images into the corresponding feature maps of real faces and the deconvolutional layers generate real faces from the transferred feature maps. To enforce the destylized faces to be similar to authentic face images, we employ a discriminative network, which consists of convolutional and fully connected layers. We demonstrate the effectiveness of our network by conducting experiments on an extensive set of synthetic images. Furthermore, we illustrate our network can recover faces from stylized portraits and real paintings for which the stylized data was unavailable during the training phase.This work is supported by the Australian Research Council (ARC) grant DP150104645

Face Recovery from Stylized Portraits using Deep Learning

Arguably, face poses form the most telling cues for nonverbal communication. Considering even straightforward and trivial interactions, humans naturally gravitate towards the face, often seeking vital information that is revealed by observing facial expressions. The visual information obtained from faces can be very valuable and interesting, for instance, many applications including the content enhancement and forensics require significant magnification of photorealistic face images. Computer analysis and photorealistic multimedia content editing are examples of applications that use face images. However, face perception at a reasonable level of accuracy is possible only if sufficient details exist in these images. Artistic portraits present further challenges as there is little scope to infer anything about underlying subjects. This thesis addresses this deficiency by presenting approaches for solving the challenge of reconstructing photorealistic images from their artistic counterparts.Specifically, the problem we consider is to recover faces from artistic portraits(also known as face destylization) including the recovery of fine detail and facial features from deteriorated portraits. To tackle this problem, we consider an approach based on Deep Neural Networks (DNNs). Through three successive studies we demonstrate that face recovery is achievable, and moreover, the faces can be re-covered with high levels of accuracy. The method we develop through the course of these studies proves very powerful, and we further demonstrate this by applying itto the problem of generating high-resolution face images from very low resolution inputs. The main contribution in this thesis is the development of a generative-discriminative DNN, which has previously shown to efficiently generate realistic images. By successively improving the network we show that significantly more accurate faces can be recovered from their respective portraits. In particular, we make cumulative improvements to the DNN in three different stages corresponding to three different studies, with each study demonstrating substantial gains. To further demonstrate the efficacy of our approach, we develop a DNN specifically for the task of face frontalization and hallucination. We show that our network can generate high-quality super-resolved and frontalized face images which are visually very close to their corresponding ground-truth counterparts, thus achieving superior face hallucination performance. In summary, this thesis presents approaches based on DNNs to recover realistic faces from portraits. Our first face destylization architecture uses a pixel-wise loss in the generative part of the network. Despite it being effective at recovering faces from portraits, it is unable to do the same when portraits are misaligned with a variety of rotations and viewpoint variations. To handle this, we extended our approach in two aspects: (a) by using STNs as intermediate layers to compensate for misalignments of input portraits, and (b) incorporate an identity-preserving loss to the generative part of our network to recover the underlying identity accurately. As a third extension, in order to recover the high-frequency facial details, we incorporate auxiliary facial attributes into the extracted feature maps. In this fashion, we fuse visual and semantic information for best visual results. This also allows us to manipulate appearance details such as hair color, facial expressions, etc. We also introduce theTANN to upsample and frontalize very low resolution unaligned face images jointly in an end-to-end fashion. We have conducted an extensive experimental analysis, for each extension of the proposed DNNs, that demonstrates the superiority of our proposed methods over the current state of the art