35 research outputs found
Face Hallucination via Deep Neural Networks.
We firstly address aligned low-resolution (LR) face images (i.e. 16X16 pixels) by designing a discriminative generative network, named URDGN. URDGN is composed of two networks: a generative model and a discriminative model.
We introduce a pixel-wise L2 regularization term to the generative model and exploit the feedback of the discriminative network to make the upsampled face images more similar to real ones.
We present an end-to-end transformative discriminative neural network (TDN) devised for super-resolving unaligned tiny face images. TDN embeds spatial transformation layers to enforce local receptive fields to line-up with similar spatial supports. To upsample noisy unaligned LR face images, we propose decoder-encoder-decoder networks. A transformative discriminative decoder network is employed to upsample and denoise LR inputs simultaneously. Then we project the intermediate HR faces to aligned and noise-free LR faces by a transformative encoder network. Finally, high-quality hallucinated HR images are generated by our second decoder. Furthermore, we present an end-to-end multiscale transformative discriminative neural network (MTDN) to super-resolve unaligned LR face images of different resolutions in a unified framework.
We propose a method that explicitly incorporates structural information of faces into the face super-resolution process by using a multi-task convolutional neural network (CNN). Our method not only uses low-level information (i.e. intensity similarity), but also middle-level information (i.e. face structure) to further explore spatial constraints of facial components from LR inputs images.
We demonstrate that supplementing residual images or feature maps with additional facial attribute information can significantly reduce the ambiguity in face super-resolution. To explore this idea, we develop an attribute-embedded upsampling network. In this manner, our method is able to super-resolve LR faces by a large upscaling factor while reducing the uncertainty of one-to-many mappings remarkably.
We further push the boundaries of hallucinating a tiny, non-frontal face image to understand how much of this is possible by leveraging the availability of large datasets and deep networks. To this end, we introduce a novel Transformative Adversarial Neural Network (TANN) to jointly frontalize very LR out-of-plane rotated face images (including profile views) and aggressively super-resolve them by 8X, regardless of their original poses and without using any 3D information. Besides recovering an HR face images from an LR version, this thesis also addresses the task of restoring realistic faces from stylized portrait images, which can also be regarded as face hallucination
FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors
Face Super-Resolution (SR) is a domain-specific super-resolution problem. The
specific facial prior knowledge could be leveraged for better super-resolving
face images. We present a novel deep end-to-end trainable Face Super-Resolution
Network (FSRNet), which makes full use of the geometry prior, i.e., facial
landmark heatmaps and parsing maps, to super-resolve very low-resolution (LR)
face images without well-aligned requirement. Specifically, we first construct
a coarse SR network to recover a coarse high-resolution (HR) image. Then, the
coarse HR image is sent to two branches: a fine SR encoder and a prior
information estimation network, which extracts the image features, and
estimates landmark heatmaps/parsing maps respectively. Both image features and
prior information are sent to a fine SR decoder to recover the HR image. To
further generate realistic faces, we propose the Face Super-Resolution
Generative Adversarial Network (FSRGAN) to incorporate the adversarial loss
into FSRNet. Moreover, we introduce two related tasks, face alignment and
parsing, as the new evaluation metrics for face SR, which address the
inconsistency of classic metrics w.r.t. visual perception. Extensive benchmark
experiments show that FSRNet and FSRGAN significantly outperforms state of the
arts for very LR face SR, both quantitatively and qualitatively. Code will be
made available upon publication.Comment: Chen and Tai contributed equally to this pape
Hallucinating very low-resolution and obscured face images
Most of the face hallucination methods are designed for complete inputs. They
will not work well if the inputs are very tiny or contaminated by large
occlusion. Inspired by this fact, we propose an obscured face hallucination
network(OFHNet). The OFHNet consists of four parts: an inpainting network, an
upsampling network, a discriminative network, and a fixed facial landmark
detection network. The inpainting network restores the low-resolution(LR)
obscured face images. The following upsampling network is to upsample the
output of inpainting network. In order to ensure the generated
high-resolution(HR) face images more photo-realistic, we utilize the
discriminative network and the facial landmark detection network to better the
result of upsampling network. In addition, we present a semantic structure
loss, which makes the generated HR face images more pleasing. Extensive
experiments show that our framework can restore the appealing HR face images
from 1/4 missing area LR face images with a challenging scaling factor of 8x.Comment: 20 pages, Submitted to Pattern Recognition Letter
Recovering Faces from Portraits with Auxiliary Facial Attributes
Recovering a photorealistic face from an artistic portrait is a challenging
task since crucial facial details are often distorted or completely lost in
artistic compositions. To handle this loss, we propose an Attribute-guided Face
Recovery from Portraits (AFRP) that utilizes a Face Recovery Network (FRN) and
a Discriminative Network (DN). FRN consists of an autoencoder with residual
block-embedded skip-connections and incorporates facial attribute vectors into
the feature maps of input portraits at the bottleneck of the autoencoder. DN
has multiple convolutional and fully-connected layers, and its role is to
enforce FRN to generate authentic face images with corresponding facial
attributes dictated by the input attribute vectors. %Leveraging on the spatial
transformer networks, FRN automatically compensates for misalignments of
portraits. % and generates aligned face images. For the preservation of
identities, we impose the recovered and ground-truth faces to share similar
visual features. Specifically, DN determines whether the recovered image looks
like a real face and checks if the facial attributes extracted from the
recovered image are consistent with given attributes. %Our method can recover
high-quality photorealistic faces from unaligned portraits while preserving the
identity of the face images as well as it can reconstruct a photorealistic face
image with a desired set of attributes. Our method can recover photorealistic
identity-preserving faces with desired attributes from unseen stylized
portraits, artistic paintings, and hand-drawn sketches. On large-scale
synthesized and sketch datasets, we demonstrate that our face recovery method
achieves state-of-the-art results.Comment: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV
Facial Attribute Capsules for Noise Face Super Resolution
Existing face super-resolution (SR) methods mainly assume the input image to
be noise-free. Their performance degrades drastically when applied to
real-world scenarios where the input image is always contaminated by noise. In
this paper, we propose a Facial Attribute Capsules Network (FACN) to deal with
the problem of high-scale super-resolution of noisy face image. Capsule is a
group of neurons whose activity vector models different properties of the same
entity. Inspired by the concept of capsule, we propose an integrated
representation model of facial information, which named Facial Attribute
Capsule (FAC). In the SR processing, we first generated a group of FACs from
the input LR face, and then reconstructed the HR face from this group of FACs.
Aiming to effectively improve the robustness of FAC to noise, we generate FAC
in semantic, probabilistic and facial attributes manners by means of integrated
learning strategy. Each FAC can be divided into two sub-capsules: Semantic
Capsule (SC) and Probabilistic Capsule (PC). Them describe an explicit facial
attribute in detail from two aspects of semantic representation and probability
distribution. The group of FACs model an image as a combination of facial
attribute information in the semantic space and probabilistic space by an
attribute-disentangling way. The diverse FACs could better combine the face
prior information to generate the face images with fine-grained semantic
attributes. Extensive benchmark experiments show that our method achieves
superior hallucination results and outperforms state-of-the-art for very low
resolution (LR) noise face image super resolution.Comment: To appear in AAAI 202