High-Quality Facial Photo-Sketch Synthesis Using Multi-Adversarial Networks

Synthesizing face sketches from real photos and its inverse have many applications. However, photo/sketch synthesis remains a challenging problem due to the fact that photo and sketch have different characteristics. In this work, we consider this task as an image-to-image translation problem and explore the recently popular generative models (GANs) to generate high-quality realistic photos from sketches and sketches from photos. Recent GAN-based methods have shown promising results on image-to-image translation problems and photo-to-sketch synthesis in particular, however, they are known to have limited abilities in generating high-resolution realistic images. To this end, we propose a novel synthesis framework called Photo-Sketch Synthesis using Multi-Adversarial Networks, (PS2-MAN) that iteratively generates low resolution to high resolution images in an adversarial way. The hidden layers of the generator are supervised to first generate lower resolution images followed by implicit refinement in the network to generate higher resolution images. Furthermore, since photo-sketch synthesis is a coupled/paired translation problem, we leverage the pair information using CycleGAN framework. Both Image Quality Assessment (IQA) and Photo-Sketch Matching experiments are conducted to demonstrate the superior performance of our framework in comparison to existing state-of-the-art solutions. Code available at: https://github.com/lidan1/PhotoSketchMAN.Comment: Accepted by 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018)(Oral

Cross Domain Face Synthesis

Cross domain face synthesis refers to the problem of synthesize faces across different domains, for example, forensic sketches vs. digital photograph, visual light vs. thermal and faces with various attributes. It has a wide range of applications from law enforcement to digital entertainment. However, cross domain synthesis remains a challenging problem due to the fact that images in different domains have different characteristics. In this thesis, we consider the task as an image-to-image translation problem and explored the recently popular generative adversarial networks (GANs) to generate high-quality realistic images. Earlier GAN-based methods have shown promising results on image-to-image translation problems, however, they are known to have limited abilities in generating high-resolution realistic images. To this end, we proposed a novel synthesis framework that iteratively generates low resolution to high resolution images in an adversarial way. The hidden layers of the generator are supervised to first generate lower resolution images followed by implicit refinement in the network to generate higher resolution images. Furthermore, since cross domain synthesis is a coupled/paired translation problem where translations at both directions are equally important, we leverage the pair information using CycleGAN framework. Evaluation of the proposed method is performed for photo-sketch synthesis problem specifically, two datasets: CUHK and CUFSF are used in this thesis. Both Image Quality Assessment (IQA) and Photo-Sketch Matching experiments are conducted to demonstrate the superior performance of our framework in comparison to existing state-of-the-art solutions. Additionally, ablation studies are conducted to verify the effectiveness iterative synthesis and various loss functions. Moreover, several future works are discussed in this thesis, including the multimodal visible to polarimetric- thermal facial image generation and attention guided image-to-image generation