3D based head movement tracking for incorporation in facial expression system

Head movement tracking is a necessary system in our attempt to establish the positioning of the head in an instance of the time. In computer graphics, head positioning sequence must be done in a proper manner so that the rendering will appear realistic. The head role becomes more important when a facial expression is being depicted. As a true facial expression must be accompanied with some motion of the head, rendering the facial expression without any proper description regarding head movement will make the head less realistic. This paper proposed a dual-pivot 3D-based head movement tracking system (DPHT) that enables modeler to capture the movement of the head. By having two pivots in the system, the movement of the neck can be modeled together with the yaw, roll and pitch of the head. This movement of the neck is an integral part of the facial expression depiction as can be attested by someone who 'pulls' his neck in manifestation of disgust. The results in this paper show that having a dual-pivot tracking system, head positioning can be better established hence producing more realistic head movement model

Model-free head pose estimation based on shape factorisation and particle filtering

This work forms part of the project Eye-Communicate funded by the Malta Council for Science and Technology through the National Research & Innovation Programme (2012) under Research Grant No. R&I-2012-057.Head pose estimation is essential for several applications and is particularly required for head pose-free eye-gaze tracking where estimation of head rotation permits free head movement during tracking. While the literature is broad, the accuracy of recent vision-based head pose estimation methods is contingent upon the availability of training data or accurate initialisation and tracking of specific facial landmarks. In this paper, we propose a method to estimate the head pose in real time from the trajectories of a set of feature points spread randomly over the face region, without requiring a training phase or model-fitting of specific facial features. Conversely, without seeking specific facial landmarks, our method exploits the sparse 3-dimensional shape of the surface of interest, recovered via shape and motion factorisation, in combination with particle filtering to correct mistracked feature points and improve upon an initial estimation of the 3-dimensional shape during tracking. In comparison with two additional methods, quantitative results obtained through our model- and landmark-free method yield a reduction in the head pose estimation error for a wide range of head rotation angles.peer-reviewe

Modelling of head movement in expression of disgust

Head movement modelling can be seen as a part of facial expression study because some expressions like disgust involves head movement. Head movement information can be acquired by video recording process. The recording process has to deal with image distortion correctable via plumb-line method. Unfortunately the linear fitting used in plumb-line requires piecewise function. The thesis aims to enhance the plumb-line-based image distortion correction using conic function coefficient evaluation replacing linear fitting. Experiments conducted shows that the proposed method handles various line orientations without having to rely on piecewise function. Besides distortion correction, an approach for expression movement tracking is needed. Optical flow-template matching is one of the techniques used for tracking. However, existing search algorithms did not discuss much on the looping technique of template matching. Moreover, tracking transient features during expression requires special process as the feature exists intermittently. The thesis aims to enhance the optical flow-template matching-based tracking method for tracking feature points during head movement by controlling the search loop and introducing anchoring to handle transient components. Experiment showed that the proposed method recorded a reduction in comparison of 40.1% over another similar method during worse case scenario. Besides reduction, the proposed method also lowered the lost point during searching when compared with existing method. Head movement modelling is not given proper attention in facial expression study hence affecting head model believability in computer graphics. The thesis aims to design head movement quantification method for head movement during disgust expression. The quantification method tracks movements of the head inclusive of the neck and named as ‘Dual Pivot Head Tracking’ (DPHT). To prove that it is perceptually better to use the proposed method, a perceptual study of expression with and without head movement was conducted. Results showed that subjects perceived disgust expression better if the proposed method is used ( -score of neck given head=14.9 vs. head given neck=3.59). To further support our proposal on the need to track head movement inclusive of the neck, experiments tracking subjects depicting disgust were conducted. A statistical two-tailed test to evaluate the existence of neck motion during head movement was done. Furthermore, visual comparison was made with a model without head movement approach. Results showed that neck motion was presence during head movement of disgust (z-score = 3.4 with p-value = 0.0006). Similarly the visual depictions showed that without the head movement inclusive of neck the rendering seemed to be incomplete. Having movement information, the thesis aims to design a temporal model of head movement during disgust expression. Neck motion, a part of head motion, plays a role during disgust expression. The thesis proposes spline-based function named Joint Cubic Bezier (JCB) to model neck motion during disgust. Experiments showed that using JCB, analysis and synthesis of neck motion during disgust expression is better than via cosine and exponential approach with angular separation score of JCB=0.986041, Exponential=0.897163 and Cosine=0.90773

SpecTracle: Wearable Facial Motion Tracking from Unobtrusive Peripheral Cameras

Facial motion tracking in head-mounted displays (HMD) has the potential to enable immersive "face-to-face" interaction in a virtual environment. However, current works on facial tracking are not suitable for unobtrusive augmented reality (AR) glasses or do not have the ability to track arbitrary facial movements. In this work, we demonstrate a novel system called SpecTracle that tracks a user's facial motions using two wide-angle cameras mounted right next to the visor of a Hololens. Avoiding the usage of cameras extended in front of the face, our system greatly improves the feasibility to integrate full-face tracking into a low-profile form factor. We also demonstrate that a neural network-based model processing the wide-angle cameras can run in real-time at 24 frames per second (fps) on a mobile GPU and track independent facial movement for different parts of the face with a user-independent model. Using a short personalized calibration, the system improves its tracking performance by 42.3% compared to the user-independent model

HeadOn: Real-time Reenactment of Human Portrait Videos

We propose HeadOn, the first real-time source-to-target reenactment approach for complete human portrait videos that enables transfer of torso and head motion, face expression, and eye gaze. Given a short RGB-D video of the target actor, we automatically construct a personalized geometry proxy that embeds a parametric head, eye, and kinematic torso model. A novel real-time reenactment algorithm employs this proxy to photo-realistically map the captured motion from the source actor to the target actor. On top of the coarse geometric proxy, we propose a video-based rendering technique that composites the modified target portrait video via view- and pose-dependent texturing, and creates photo-realistic imagery of the target actor under novel torso and head poses, facial expressions, and gaze directions. To this end, we propose a robust tracking of the face and torso of the source actor. We extensively evaluate our approach and show significant improvements in enabling much greater flexibility in creating realistic reenacted output videos.Comment: Video: https://www.youtube.com/watch?v=7Dg49wv2c_g Presented at Siggraph'1

Facial feature point tracking based on a graphical model framework

In this thesis a facial feature point tracker that can be used in applications such as human-computer interfaces, facial expression analysis systems, driver fatigue detection systems, etc. is proposed. The proposed tracker is based on a graphical model framework. The position of the facial features are tracked through video streams by incorporating statistical relations in time and the spatial relations between feature points. In many application areas, including those mentioned above, tracking is a key intermediate step that has a significant effect on the overall system performance. For this reason, a good practical tracking algorithm should take into account real-world phenomena such as arbitrary head movements and occlusions. Many existing algorithms track each feature point independently, and do not properly handle occlusions. This causes drifts in the case of arbitrary head movements and occlusions. By exploiting the spatial relationships between feature points, the proposed method provides robustness in a number of scenarios, including e.g. various head movements. To prevent drifts because of occlusions, a Gabor feature based occlusion detector is developed and used in the proposed method. The performance of the proposed tracker has been evaluated on real video data under various conditions. These conditions include occluded facial gestures, low video resolution, illumination changes in the scene, in-plane head motion, and out-of-plane head motion. The proposed method has also been tested on videos recorded in a vehicle environment, in order to evaluate its performance in a practical setting. Given these results it can be concluded that the proposed method provides a general promising framework for facial feature tracking. It is a robust tracker for facial expression sequences in which there are occlusions and arbitrary head movements. The results in the vehicle environment suggest that the proposed method has the potential to be useful for tasks such as driver behavior analysis or driver fatigue detection

A graphical model based solution to the facial feature point tracking problem

In this paper a facial feature point tracker that is motivated by applications such as human-computer interfaces and facial expression analysis systems is proposed. The proposed tracker is based on a graphical model framework. The facial features are tracked through video streams by incorporating statistical relations in time as well as spatial relations between feature points. By exploiting the spatial relationships between feature points, the proposed method provides robustness in real-world conditions such as arbitrary head movements and occlusions. A Gabor feature-based occlusion detector is developed and used to handle occlusions. The performance of the proposed tracker has been evaluated on real video data under various conditions including occluded facial gestures and head movements. It is also compared to two popular methods, one based on Kalman filtering exploiting temporal relations, and the other based on active appearance models (AAM). Improvements provided by the proposed approach are demonstrated through both visual displays and quantitative analysis

Head Tracking via Robust Registration in Texture Map Images

A novel method for 3D head tracking in the presence of large head rotations and facial expression changes is described. Tracking is formulated in terms of color image registration in the texture map of a 3D surface model. Model appearance is recursively updated via image mosaicking in the texture map as the head orientation varies. The resulting dynamic texture map provides a stabilized view of the face that can be used as input to many existing 2D techniques for face recognition, facial expressions analysis, lip reading, and eye tracking. Parameters are estimated via a robust minimization procedure; this provides robustness to occlusions, wrinkles, shadows, and specular highlights. The system was tested on a variety of sequences taken with low quality, uncalibrated video cameras. Experimental results are reported

3D face tracking and multi-scale, spatio-temporal analysis of linguistically significant facial expressions and head positions in ASL

Essential grammatical information is conveyed in signed languages by clusters of events involving facial expressions and movements of the head and upper body. This poses a significant challenge for computer-based sign language recognition. Here, we present new methods for the recognition of nonmanual grammatical markers in American Sign Language (ASL) based on: (1) new 3D tracking methods for the estimation of 3D head pose and facial expressions to determine the relevant low-level features; (2) methods for higher-level analysis of component events (raised/lowered eyebrows, periodic head nods and head shakes) used in grammatical markings—with differentiation of temporal phases (onset, core, offset, where appropriate), analysis of their characteristic properties, and extraction of corresponding features; (3) a 2-level learning framework to combine lowand high-level features of differing spatio-temporal scales. This new approach achieves significantly better tracking and recognition results than our previous methods

Efficient illumination independent appearance-based face tracking

One of the major challenges that visual tracking algorithms face nowadays is being able to cope with changes in the appearance of the target during tracking. Linear subspace models have been extensively studied and are possibly the most popular way of modelling target appearance. We introduce a linear subspace representation in which the appearance of a face is represented by the addition of two approxi- mately independent linear subspaces modelling facial expressions and illumination respectively. This model is more compact than previous bilinear or multilinear ap- proaches. The independence assumption notably simplifies system training. We only require two image sequences. One facial expression is subject to all possible illumina- tions in one sequence and the face adopts all facial expressions under one particular illumination in the other. This simple model enables us to train the system with no manual intervention. We also revisit the problem of efficiently fitting a linear subspace-based model to a target image and introduce an additive procedure for solving this problem. We prove that Matthews and Baker’s Inverse Compositional Approach makes a smoothness assumption on the subspace basis that is equiva- lent to Hager and Belhumeur’s, which worsens convergence. Our approach differs from Hager and Belhumeur’s additive and Matthews and Baker’s compositional ap- proaches in that we make no smoothness assumptions on the subspace basis. In the experiments conducted we show that the model introduced accurately represents the appearance variations caused by illumination changes and facial expressions. We also verify experimentally that our fitting procedure is more accurate and has better convergence rate than the other related approaches, albeit at the expense of a slight increase in computational cost. Our approach can be used for tracking a human face at standard video frame rates on an average personal computer