Visibility Constrained Generative Model for Depth-based 3D Facial Pose Tracking

In this paper, we propose a generative framework that unifies depth-based 3D facial pose tracking and face model adaptation on-the-fly, in the unconstrained scenarios with heavy occlusions and arbitrary facial expression variations. Specifically, we introduce a statistical 3D morphable model that flexibly describes the distribution of points on the surface of the face model, with an efficient switchable online adaptation that gradually captures the identity of the tracked subject and rapidly constructs a suitable face model when the subject changes. Moreover, unlike prior art that employed ICP-based facial pose estimation, to improve robustness to occlusions, we propose a ray visibility constraint that regularizes the pose based on the face model's visibility with respect to the input point cloud. Ablation studies and experimental results on Biwi and ICT-3DHP datasets demonstrate that the proposed framework is effective and outperforms completing state-of-the-art depth-based methods

Real-time 3D face localization and verification

We present a method for real-time 3D face localization and verification using a consumer-grade depth camera. Our approach consists of three parts: face detection, head pose estimation, and face verification. Face detection is performed using a standard detection framework which we significantly improve by leveraging depth information. To estimate the pose of the detected face, we developed a technique that uses a combination of the particle swarm optimization (PSO) and the iterative closest point (ICP) algorithm to accurately align a 3D face model to the measured depth data. With the face localized within the image, we can compare a database 3D face model to the depth image to verify the identity of the subject. We learn a similarity metric using a random decision forest to accurately authenticate the subject. We demonstrate state-of-the-art results for both face localization and face verification on standard datasets. Since the camera and our method operate at video rate, our system is capable of continuously authenticating a subject while he/she uses his/her device

Face-from-Depth for Head Pose Estimation on Depth Images

Depth cameras allow to set up reliable solutions for people monitoring and behavior understanding, especially when unstable or poor illumination conditions make unusable common RGB sensors. Therefore, we propose a complete framework for the estimation of the head and shoulder pose based on depth images only. A head detection and localization module is also included, in order to develop a complete end-to-end system. The core element of the framework is a Convolutional Neural Network, called POSEidon+, that receives as input three types of images and provides the 3D angles of the pose as output. Moreover, a Face-from-Depth component based on a Deterministic Conditional GAN model is able to hallucinate a face from the corresponding depth image. We empirically demonstrate that this positively impacts the system performances. We test the proposed framework on two public datasets, namely Biwi Kinect Head Pose and ICT-3DHP, and on Pandora, a new challenging dataset mainly inspired by the automotive setup. Experimental results show that our method overcomes several recent state-of-art works based on both intensity and depth input data, running in real-time at more than 30 frames per second

Random Forests for Real Time 3D Face Analysis

We present a random forest-based framework for real time head pose estimation from depth images and extend it to localize a set of facial features in 3D. Our algorithm takes a voting approach, where each patch extracted from the depth image can directly cast a vote for the head pose or each of the facial features. Our system proves capable of handling large rotations, partial occlusions, and the noisy depth data acquired using commercial sensors. Moreover, the algorithm works on each frame independently and achieves real time performance without resorting to parallel computations on a GPU. We present extensive experiments on publicly available, challenging datasets and present a new annotated head pose database recorded using a Microsoft Kinec

Real-time appearance-based gaze tracking.

PhDGaze tracking technology is widely used in Human Computer Interaction applications such as in interfaces for assisting people with disabilities and for driver attention monitoring. However, commercially available gaze trackers are expensive and their performance deteriorates if the user is not positioned in front of the camera and facing it. Also, head motion or being far from the device degrades their accuracy. This thesis focuses on the development of real-time time appearance based gaze tracking algorithms using low cost devices, such as a webcam or Kinect. The proposed algorithms are developed by considering accuracy, robustness to head pose variation and the ability to generalise to different persons. In order to deal with head pose variation, we propose to estimate the head pose and then compensate for the appearance change and the bias to a gaze estimator that it introduces. Head pose is estimated by a novel method that utilizes tensor-based regressors at the leaf nodes of a random forest. For a baseline gaze estimator we use an SVM-based appearance-based regressor. For compensating the appearance variation introduced by the head pose, we use a geometric model, and for compensating for the bias we use a regression function that has been trained on a training set. Our methods are evaluated on publicly available dataset

Registration of non-rigidly deforming objects

This thesis investigates the current state-of-the-art in registration of non-rigidly deforming shapes. In particular, the problem of non-isometry is considered. First, a method to address locally anisotropic deformation is proposed. The subsequent evaluation of this method highlights a lack of resources for evaluating such methods. Three novel registration/shape correspondence benchmark datasets are developed for assessing different aspects of non-rigid deformation. Deficiencies in current evaluative measures are identified, leading to the development of a new performance measure that effectively communicates the density and distribution of correspondences. Finally, the problem of transferring skull orbit labels between scans is examined on a database of unlabelled skulls. A novel pipeline that mitigates errors caused by coarse representations is proposed