FaceShop: Deep Sketch-based Face Image Editing

We present a novel system for sketch-based face image editing, enabling users to edit images intuitively by sketching a few strokes on a region of interest. Our interface features tools to express a desired image manipulation by providing both geometry and color constraints as user-drawn strokes. As an alternative to the direct user input, our proposed system naturally supports a copy-paste mode, which allows users to edit a given image region by using parts of another exemplar image without the need of hand-drawn sketching at all. The proposed interface runs in real-time and facilitates an interactive and iterative workflow to quickly express the intended edits. Our system is based on a novel sketch domain and a convolutional neural network trained end-to-end to automatically learn to render image regions corresponding to the input strokes. To achieve high quality and semantically consistent results we train our neural network on two simultaneous tasks, namely image completion and image translation. To the best of our knowledge, we are the first to combine these two tasks in a unified framework for interactive image editing. Our results show that the proposed sketch domain, network architecture, and training procedure generalize well to real user input and enable high quality synthesis results without additional post-processing.Comment: 13 pages, 20 figure

MODELING AND ANALYSIS OF WRINKLES ON AGING HUMAN FACES

The analysis and modeling of aging human faces has been extensively studied in the past decade. Most of this work is based on matching learning techniques focused on appearance of faces at different ages incorporating facial features such as face shape/geometry and patch-based texture features. However, we do not find much work done on the analysis of facial wrinkles in general and specific to a person. The goal of this dissertation is to analyse and model facial wrinkles for different applications. Facial wrinkles are challenging low-level image features to analyse. In general, skin texture has drastically varying appearance due to its characteristic physical properties. A skin patch looks very different when viewed or illuminated from different angles. This makes subtle skin features like facial wrinkles difficult to be detected in images acquired in uncontrolled imaging settings. In this dissertation, we examine the image properties of wrinkles i.e. intensity gradients and geometric properties and use them for several applications including low-level image processing for automatic detection/localization of wrinkles, soft biometrics and removal of wrinkles using digital inpainting. First, we present results of detection/localization of wrinkles in images using Marked Point Process (MPP). Wrinkles are modeled as sequences of line segments in a Bayesian framework which incorporates a prior probability model based on the likely geometric properties of wrinkles and a data likelihood term based on image intensity gradients. Wrinkles are localized by sampling the posterior probability using a Reversible Jump Markov Chain Monte Carlo (RJMCMC) algorithm. We also present an evaluation algorithm to quantitatively evaluate the detection and false alarm rate of our algorithm and conduct experiments with images taken in uncontrolled settings. The MPP model, despite its promising localization results, requires a large number of iterations in the RJMCMC algorithm to reach global minimum resulting in considerable computation time. This motivated us to adopt a deterministic approach based on image morphology for fast localization of facial wrinkles. We propose image features based on Gabor filter banks to highlight subtle curvilinear discontinuities in skin texture caused by wrinkles. Then, image morphology is used to incorporate geometric constraints to localize curvilinear shapes of wrinkles at image sites of large Gabor filter responses. We conduct experiments on two sets of low and high resolution images to demonstrate faster and visually better localization results as compared to those obtained by MPP modeling. As a next application, we investigate the user-drawn and automatically detected wrinkles as a pattern for their discriminative power as a soft biometrics to recognize subjects from their wrinkle patterns only. A set of facial wrinkles from an image is treated as a curve pattern and used for subject recognition. Given the wrinkle patterns from a query and gallery images, several distance measures are calculated between the two patterns to quantify the similarity between them. This is done by finding the possible correspondences between curves from the two patterns using a simple bipartite graph matching algorithm. Then several metrics are used to calculate the similarity between the two wrinkle patterns. These metrics are based on Hausdorff distance and curve-to-curve correspondences. We conduct experiments on data sets of both hand drawn and automatically detected wrinkles. Finally, we apply digital inpainting to automatically remove wrinkles from facial images. Digital image inpainting refers to filling in the holes of arbitrary shapes in images so that they seem to be part of the original image. The inpainting methods target either the structure or the texture of an image or both. There are two limitations of existing inpainting methods for the removal of wrinkles. First, the differences in the attributes of structure and texture requires different inpainting methods. Facial wrinkles do not fall strictly under the category of structure or texture and can be considered as some where in between. Second, almost all of the image inpainting techniques are supervised i.e. the area/gap to be filled is provided by user interaction and the algorithms attempt to find the suitable image portion automatically. We present an unsupervised image inpainting method where facial regions with wrinkles are detected automatically using their characteristic intensity gradients and removed by painting the regions by the surrounding skin texture

Robust Specularity Removal from Hand-held Videos

Specular reflection exists when one tries to record a photo or video through a transparent glass medium or opaque surfaces such as plastics, ceramics, polyester and human skin, which can be well described as the superposition of a transmitted layer and a reflection layer. These specular reflections often confound the algorithms developed for image analysis, computer vision and pattern recognition. To obtain a pure diffuse reflection component, specularity (highlights) needs to be removed. To handle this problem, a novel and robust algorithm is formulated. The contributions of this work are three-fold.;First, the smoothness of the video along with the temporal coherence and illumination changes are preserved by reducing the flickering and jagged edges caused by hand-held video acquisition and homography transformation respectively.;Second, this algorithm is designed to improve upon the state-of-art algorithms by automatically selecting the region of interest (ROI) for all the frames, reducing the computational time and complexity by utilizing the luminance (Y) channel and exploiting the Augmented Lagrange Multiplier (ALM) with Alternating Direction Minimizing (ADM) to facilitate the derivation of solution algorithms.;Third, a quantity metrics is devised, which objectively quantifies the amount of specularity in each frame of a hand-held video. The proposed specularity removal algorithm is compared against existing state-of-art algorithms using the newly-developed quantity metrics. Experimental results validate that the developed algorithm has superior performance in terms of computation time, quality and accuracy

A survey of face recognition techniques under occlusion

The limited capacity to recognize faces under occlusions is a long-standing problem that presents a unique challenge for face recognition systems and even for humans. The problem regarding occlusion is less covered by research when compared to other challenges such as pose variation, different expressions, etc. Nevertheless, occluded face recognition is imperative to exploit the full potential of face recognition for real-world applications. In this paper, we restrict the scope to occluded face recognition. First, we explore what the occlusion problem is and what inherent difficulties can arise. As a part of this review, we introduce face detection under occlusion, a preliminary step in face recognition. Second, we present how existing face recognition methods cope with the occlusion problem and classify them into three categories, which are 1) occlusion robust feature extraction approaches, 2) occlusion aware face recognition approaches, and 3) occlusion recovery based face recognition approaches. Furthermore, we analyze the motivations, innovations, pros and cons, and the performance of representative approaches for comparison. Finally, future challenges and method trends of occluded face recognition are thoroughly discussed

Image-based Material Editing

Photo editing software allows digital images to be blurred, warped or re-colored at the touch of a button. However, it is not currently possible to change the material appearance of an object except by painstakingly painting over the appropriate pixels. Here we present a set of methods for automatically replacing one material with another, completely different material, starting with only a single high dynamic range image, and an alpha matte specifying the object. Our approach exploits the fact that human vision is surprisingly tolerant of certain (sometimes enormous) physical inaccuracies. Thus, it may be possible to produce a visually compelling illusion of material transformations, without fully reconstructing the lighting or geometry. We employ a range of algorithms depending on the target material. First, an approximate depth map is derived from the image intensities using bilateral filters. The resulting surface normals are then used to map data onto the surface of the object to specify its material appearance. To create transparent or translucent materials, the mapped data are derived from the object\u27s background. To create textured materials, the mapped data are a texture map. The surface normals can also be used to apply arbitrary bidirectional reflectance distribution functions to the surface, allowing us to simulate a wide range of materials. To facilitate the process of material editing, we generate the HDR image with a novel algorithm, that is robust against noise in individual exposures. This ensures that any noise, which would possibly have affected the shape recovery of the objects adversely, will be removed. We also present an algorithm to automatically generate alpha mattes. This algorithm requires as input two images--one where the object is in focus, and one where the background is in focus--and then automatically produces an approximate matte, indicating which pixels belong to the object. The result is then improved by a second algorithm to generate an accurate alpha matte, which can be given as input to our material editing techniques

3D Human Face Reconstruction and 2D Appearance Synthesis

3D human face reconstruction has been an extensive research for decades due to its wide applications, such as animation, recognition and 3D-driven appearance synthesis. Although commodity depth sensors are widely available in recent years, image based face reconstruction are significantly valuable as images are much easier to access and store. In this dissertation, we first propose three image-based face reconstruction approaches according to different assumption of inputs. In the first approach, face geometry is extracted from multiple key frames of a video sequence with different head poses. The camera should be calibrated under this assumption. As the first approach is limited to videos, we propose the second approach then focus on single image. This approach also improves the geometry by adding fine grains using shading cue. We proposed a novel albedo estimation and linear optimization algorithm in this approach. In the third approach, we further loose the constraint of the input image to arbitrary in the wild images. Our proposed approach can robustly reconstruct high quality model even with extreme expressions and large poses. We then explore the applicability of our face reconstructions on four interesting applications: video face beautification, generating personalized facial blendshape from image sequences, face video stylizing and video face replacement. We demonstrate great potentials of our reconstruction approaches on these real-world applications. In particular, with the recent surge of interests in VR/AR, it is increasingly common to see people wearing head-mounted displays. However, the large occlusion on face is a big obstacle for people to communicate in a face-to-face manner. Our another application is that we explore hardware/software solutions for synthesizing the face image with presence of HMDs. We design two setups (experimental and mobile) which integrate two near IR cameras and one color camera to solve this problem. With our algorithm and prototype, we can achieve photo-realistic results. We further propose a deep neutral network to solve the HMD removal problem considering it as a face inpainting problem. This approach doesn\u27t need special hardware and run in real-time with satisfying results