Style transfer for headshot portraits

Headshot portraits are a popular subject in photography but to achieve a compelling visual style requires advanced skills that a casual photographer will not have. Further, algorithms that automate or assist the stylization of generic photographs do not perform well on headshots due to the feature-specific, local retouching that a professional photographer typically applies to generate such portraits. We introduce a technique to transfer the style of an example headshot photo onto a new one. This can allow one to easily reproduce the look of renowned artists. At the core of our approach is a new multiscale technique to robustly transfer the local statistics of an example portrait onto a new one. This technique matches properties such as the local contrast and the overall lighting direction while being tolerant to the unavoidable differences between the faces of two different people. Additionally, because artists sometimes produce entire headshot collections in a common style, we show how to automatically find a good example to use as a reference for a given portrait, enabling style transfer without the user having to search for a suitable example for each input. We demonstrate our approach on data taken in a controlled environment as well as on a large set of photos downloaded from the Internet. We show that we can successfully handle styles by a variety of different artists.Quanta Computer (Firm)Adobe System

HeadOn: Real-time Reenactment of Human Portrait Videos

We propose HeadOn, the first real-time source-to-target reenactment approach for complete human portrait videos that enables transfer of torso and head motion, face expression, and eye gaze. Given a short RGB-D video of the target actor, we automatically construct a personalized geometry proxy that embeds a parametric head, eye, and kinematic torso model. A novel real-time reenactment algorithm employs this proxy to photo-realistically map the captured motion from the source actor to the target actor. On top of the coarse geometric proxy, we propose a video-based rendering technique that composites the modified target portrait video via view- and pose-dependent texturing, and creates photo-realistic imagery of the target actor under novel torso and head poses, facial expressions, and gaze directions. To this end, we propose a robust tracking of the face and torso of the source actor. We extensively evaluate our approach and show significant improvements in enabling much greater flexibility in creating realistic reenacted output videos.Comment: Video: https://www.youtube.com/watch?v=7Dg49wv2c_g Presented at Siggraph'1

Stylizing Face Images via Multiple Exemplars

We address the problem of transferring the style of a headshot photo to face images. Existing methods using a single exemplar lead to inaccurate results when the exemplar does not contain sufficient stylized facial components for a given photo. In this work, we propose an algorithm to stylize face images using multiple exemplars containing different subjects in the same style. Patch correspondences between an input photo and multiple exemplars are established using a Markov Random Field (MRF), which enables accurate local energy transfer via Laplacian stacks. As image patches from multiple exemplars are used, the boundaries of facial components on the target image are inevitably inconsistent. The artifacts are removed by a post-processing step using an edge-preserving filter. Experimental results show that the proposed algorithm consistently produces visually pleasing results.Comment: In CVIU 2017. Project Page: http://www.cs.cityu.edu.hk/~yibisong/cviu17/index.htm

Multi-Class Semantic Segmentation of Faces

In this paper the problem of multi-class face segmentation is introduced. Differently from previous works which only consider few classes - typically skin and hair - the label set is extended here to six categories: skin, hair, eyes, nose, mouth and background. A dataset with 70 images taken from MIT-CBCL and FEI face databases is manually annotated and made publicly available1. Three kind of local features - accounting for color, shape and location - are extracted from uniformly sampled square patches. A discriminative model is built with random decision forests and used for classification. Many different combinations of features and parameters are explored to find the best possible model configuration. Our analysis shows that very good performance (~ 93% in accuracy) can be achieved with a fairly simple model

Scene Determination based on Video and Audio Features

Determination of scenes from a video is a challenging task. When asking humans for it, results will be inconsistent since the term scene is not precisely defined. It leaves it up to each human to set shared attributes which integrate shots to scenes. However, consistent results can be found for certain basic attributes like dialogs, same settings and continuing sounds. We have therefore developed a scene determination scheme which clusters shots based on detected dialogs, same settings and similar audio. Our experimental results show that automatic deter mination of these types of scenes can be performed reliably

Non-photorealistic rendering of portraits

We describe an image-based non-photorealistic rendering pipeline for creating portraits in two styles: The first is a somewhat “puppet” like rendering, that treats the face like a relatively uniform smooth surface, with the geometry being emphasised by shading. The second style is inspired by the artist Julian Opie, in which the human face is reduced to its essentials, i.e. homogeneous skin, thick black lines, and facial features such as eyes and the nose represented in a cartoon manner. Our method is able to automatically generate these stylisations without requiring the input images to be tightly cropped, direct frontal view, and moreover perform abstraction while maintaining the distinctiveness of the portraits (i.e. they should remain recognisable)

(SEMI)-AUTOMATED ANALYSIS OF MELANOCYTIC LESIONS

Melanoma is a very aggressive form of skin cancer whose incidence has constantly grown in the last 50 years. To increase the survival rate, an early diagnosis followed by a prompt excision is crucial and requires an accurate and periodic analysis of the patient's melanocytic lesions. We have developed an hardware and software solution named Mole Mapper to assist the dermatologists during the diagnostic process. The goal is to increase the accuracy of the diagnosis, accelerating the entire process at the same time. This is achieved through an automated analysis of the dermatoscopic images which computes and highlights the proper information to the dermatologist. In this thesis we present the 3 main algorithms that have been implemented into the Mole Mapper: A robust segmentation of the melanocytic lesion, which is the starting point for any other image processing algorithm and which allows the extraction of useful information about the lesion's shape and size. It outperforms the speed and quality of other state-of-the-art methods, with a precision that meets a Senior Dermatologist's standard and an execution time that allows for real-time video processing; A virtual shaving algorithm, which increases the precision and robustness of the other computer vision algorithms and provides the dermatologist with a hair-free image to be used during the evaluation process. It matches the quality of state-of-the-art methods but requires only a fraction of the computational time, allowing for computation on a mobile device in a time-frame compatible with an interactive GUI; A registration algorithm through which to study the evolution of the lesion over time, highlighting any unexpected anomalies and variations. Since a standard approach to this problem has not yet been proposed, we define the scope and constraints of the problem; we analyze the results and issues of standard registration techniques; and finally, we propose an algorithm with a speed compatible with Mole Mapper's constraints and with an accuracy comparable to the registration performed by a human operator