Human Recognition from Video Sequences and Off-Angle Face Images Supported by Respiration Signatures

In this work, we study the problem of human identity recognition using human respiratory waveforms extracted from videos combined with component-based off- angle human facial images. Our proposed system is composed of (i) a physiology- based human clustering module and (ii) an identification module based on facial features (nose, mouth, etc.) fetched from face videos. In our proposed methodology we, first, manage to passively extract an important vital sign (breath), cluster human subjects into nostril motion vs. nostril non-motion groups, and, then, localize a set of facial features, before we apply feature extraction and matching.;Our novel human identity recognition system is very robust, since it is working well when dealing with breath signals and a combination of different facial components acquired in uncontrolled luminous conditions. This is achieved by using our proposed Motion Classification approach and Feature Clustering technique based on the breathing waveforms we produce. The contributions of this work are three-fold. First, we collected a set of different datasets where we tested our proposed approach. Specifically, we considered six different types of facial components and their combination, to generate face-based video datasets, which present two diverse data collection conditions, i.e. videos acquired in head fully frontal position (baseline) and head looking up pose. Second, we propose a new way of passively measuring human breath from face videos and show comparatively identical output against baseline breathing waveforms produced by an ADInstruments device. Third, we demonstrate good human recognition performance when using the pro- posed pre-processing procedure of Motion Classification and Feature Clustering, working on partial features of human faces.;Our method achieves increased identification rates across all datasets used, and it manages to obtain a significantly high identification rate (ranging from 96%-100% when using a single or a combination of facial features), yielding an average of 7% raise, when compared to the baseline scenario. To the best of our knowledge, this is the first time that a biometric system is composed of an important human vital sign (breath) that is fused with facial features is such an efficient manner

3D Human Face Reconstruction and 2D Appearance Synthesis

3D human face reconstruction has been an extensive research for decades due to its wide applications, such as animation, recognition and 3D-driven appearance synthesis. Although commodity depth sensors are widely available in recent years, image based face reconstruction are significantly valuable as images are much easier to access and store. In this dissertation, we first propose three image-based face reconstruction approaches according to different assumption of inputs. In the first approach, face geometry is extracted from multiple key frames of a video sequence with different head poses. The camera should be calibrated under this assumption. As the first approach is limited to videos, we propose the second approach then focus on single image. This approach also improves the geometry by adding fine grains using shading cue. We proposed a novel albedo estimation and linear optimization algorithm in this approach. In the third approach, we further loose the constraint of the input image to arbitrary in the wild images. Our proposed approach can robustly reconstruct high quality model even with extreme expressions and large poses. We then explore the applicability of our face reconstructions on four interesting applications: video face beautification, generating personalized facial blendshape from image sequences, face video stylizing and video face replacement. We demonstrate great potentials of our reconstruction approaches on these real-world applications. In particular, with the recent surge of interests in VR/AR, it is increasingly common to see people wearing head-mounted displays. However, the large occlusion on face is a big obstacle for people to communicate in a face-to-face manner. Our another application is that we explore hardware/software solutions for synthesizing the face image with presence of HMDs. We design two setups (experimental and mobile) which integrate two near IR cameras and one color camera to solve this problem. With our algorithm and prototype, we can achieve photo-realistic results. We further propose a deep neutral network to solve the HMD removal problem considering it as a face inpainting problem. This approach doesn\u27t need special hardware and run in real-time with satisfying results

Masks: Maintaining Anonymity by Sequestering Key Statistics

High-resolution digital cameras are becoming ever-larger parts of our daily lives, whether as part of closed-circuit surveillance systems or as part of portable digital devices that many of us carry around with us. Combining the broadening reach of these cameras with automatic face recognition technology creates a sensor network that is ripe for abuse: our every action could be recorded and tagged with our identities, the date, and our location as if we each had an investigator tasked only with keeping each of us under constant surveillance. Adding the continually falling cost of data storage to this mix, and we are left with a situation where the privacy abuses don\u27t need to happen today: the stored imagery can be mined and re-mined forever, while the sophistication of automatic analysis continues to grow. The MASKS project takes the first steps toward addressing this problem. If we would like to be able to de-identify faces before the images are shared with others, we cannot do so with ad hoc techniques applied identically to all faces. Since each face is unique, the method of disguising that face must be equally unique. In order to hide or reduce those critical identifying characteristics, we are delivering the following foundational contributions toward characterizing the nature of facial information: - We have created a new pose-controlled, high-resolution database of facial images. - The most prominent anatomical markers on each face have been marked for position and shape, establishing a new gold standard for facial segmentation. - A parameterized model of the diversity of our subject population was built based on statistical analysis of the annotations. The model was validated by comparison with the performance of a standard set of artificial disguises