From 3D Point Clouds to Pose-Normalised Depth Maps

We consider the problem of generating either pairwise-aligned or pose-normalised depth maps from noisy 3D point clouds in a relatively unrestricted poses. Our system is deployed in a 3D face alignment application and consists of the following four stages: (i) data filtering, (ii) nose tip identification and sub-vertex localisation, (iii) computation of the (relative) face orientation, (iv) generation of either a pose aligned or a pose normalised depth map. We generate an implicit radial basis function (RBF) model of the facial surface and this is employed within all four stages of the process. For example, in stage (ii), construction of novel invariant features is based on sampling this RBF over a set of concentric spheres to give a spherically-sampled RBF (SSR) shape histogram. In stage (iii), a second novel descriptor, called an isoradius contour curvature signal, is defined, which allows rotational alignment to be determined using a simple process of 1D correlation. We test our system on both the University of York (UoY) 3D face dataset and the Face Recognition Grand Challenge (FRGC) 3D data. For the more challenging UoY data, our SSR descriptors significantly outperform three variants of spin images, successfully identifying nose vertices at a rate of 99.6%. Nose localisation performance on the higher quality FRGC data, which has only small pose variations, is 99.9%. Our best system successfully normalises the pose of 3D faces at rates of 99.1% (UoY data) and 99.6% (FRGC data)

Anatomical curve identification

Methods for capturing images in three dimensions are now widely available, with stereo-photogrammetry and laser scanning being two common approaches. In anatomical studies, a number of landmarks are usually identified manually from each of these images and these form the basis of subsequent statistical analysis. However, landmarks express only a very small proportion of the information available from the images. Anatomically defined curves have the advantage of providing a much richer expression of shape. This is explored in the context of identifying the boundary of breasts from an image of the female torso and the boundary of the lips from a facial image. The curves of interest are characterised by ridges or valleys. Key issues in estimation are the ability to navigate across the anatomical surface in three-dimensions, the ability to recognise the relevant boundary and the need to assess the evidence for the presence of the surface feature of interest. The first issue is addressed by the use of principal curves, as an extension of principal components, the second by suitable assessment of curvature and the third by change-point detection. P-spline smoothing is used as an integral part of the methods but adaptations are made to the specific anatomical features of interest. After estimation of the boundary curves, the intermediate surfaces of the anatomical feature of interest can be characterised by surface interpolation. This allows shape variation to be explored using standard methods such as principal components. These tools are applied to a collection of images of women where one breast has been reconstructed after mastectomy and where interest lies in shape differences between the reconstructed and unreconstructed breasts. They are also applied to a collection of lip images where possible differences in shape between males and females are of interest

Final Report to NSF of the Standards for Facial Animation Workshop

The human face is an important and complex communication channel. It is a very familiar and sensitive object of human perception. The facial animation field has increased greatly in the past few years as fast computer graphics workstations have made the modeling and real-time animation of hundreds of thousands of polygons affordable and almost commonplace. Many applications have been developed such as teleconferencing, surgery, information assistance systems, games, and entertainment. To solve these different problems, different approaches for both animation control and modeling have been developed

The Digital Bee Brain: Integrating and Managing Neurons in a Common 3D Reference System

The honeybee standard brain (HSB) serves as an interactive tool for relating morphologies of bee brain neurons and provides a reference system for functional and bibliographical properties (http://www.neurobiologie.fu-berlin.de/beebrain/). The ultimate goal is to document not only the morphological network properties of neurons collected from separate brains, but also to establish a graphical user interface for a neuron-related data base. Here, we review the current methods and protocols used to incorporate neuronal reconstructions into the HSB. Our registration protocol consists of two separate steps applied to imaging data from two-channel confocal microscopy scans: (1) The reconstruction of the neuron, facilitated by an automatic extraction of the neuron's skeleton based on threshold segmentation, and (2) the semi-automatic 3D segmentation of the neuropils and their registration with the HSB. The integration of neurons in the HSB is performed by applying the transformation computed in step (2) to the reconstructed neurons of step (1). The most critical issue of this protocol in terms of user interaction time – the segmentation process – is drastically improved by the use of a model-based segmentation process. Furthermore, the underlying statistical shape models (SSM) allow the visualization and analysis of characteristic variations in large sets of bee brain data. The anatomy of neural networks composed of multiple neurons that are registered into the HSB are visualized by depicting the 3D reconstructions together with semantic information with the objective to integrate data from multiple sources (electrophysiology, imaging, immunocytochemistry, molecular biology). Ultimately, this will allow the user to specify cell types and retrieve their morphologies along with physiological characterizations

Computational Multimedia for Video Self Modeling

Video self modeling (VSM) is a behavioral intervention technique in which a learner models a target behavior by watching a video of oneself. This is the idea behind the psychological theory of self-efficacy - you can learn or model to perform certain tasks because you see yourself doing it, which provides the most ideal form of behavior modeling. The effectiveness of VSM has been demonstrated for many different types of disabilities and behavioral problems ranging from stuttering, inappropriate social behaviors, autism, selective mutism to sports training. However, there is an inherent difficulty associated with the production of VSM material. Prolonged and persistent video recording is required to capture the rare, if not existed at all, snippets that can be used to string together in forming novel video sequences of the target skill. To solve this problem, in this dissertation, we use computational multimedia techniques to facilitate the creation of synthetic visual content for self-modeling that can be used by a learner and his/her therapist with a minimum amount of training data. There are three major technical contributions in my research. First, I developed an Adaptive Video Re-sampling algorithm to synthesize realistic lip-synchronized video with minimal motion jitter. Second, to denoise and complete the depth map captured by structure-light sensing systems, I introduced a layer based probabilistic model to account for various types of uncertainties in the depth measurement. Third, I developed a simple and robust bundle-adjustment based framework for calibrating a network of multiple wide baseline RGB and depth cameras

Shape curve analysis using curvature

Statistical shape analysis is a field for which there is growing demand. One of the major drivers for this growth is the number of practical applications which can use statistical shape analysis to provide useful insight. An example of one of these practical applications is investigating and comparing facial shapes. An ever improving suite of digital imaging technology can capture data on the three-dimensional shape of facial features from standard images. A field for which this offers a large amount of potential analytical benefit is the reconstruction of the facial surface of children born with a cleft lip or a cleft lip and palate. This thesis will present two potential methods for analysing data on the facial shape of children who were born with a cleft lip and/or palate using data from two separate studies. One form of analysis will compare the facial shape of one year old children born with a cleft lip and/or palate with the facial shape of control children. The second form of analysis will look for relationships between facial shape and psychological score for ten year old children born with a cleft lip and/or palate. While many of the techniques in this thesis could be extended to different applications much of the work is carried out with the express intention of producing meaningful analysis of the cleft children studies. Shape data can be defined as the information remaining to describe the shape of an object after removing the effects of location, rotation and scale. There are numerous techniques in the literature to remove the effects of location, rotation and scale and thereby define and compare the shapes of objects. A method which does not require the removal of the effects of location and rotation is to define the shape according to the bending of important shape curves. This method can naturally provide a technique for investigating facial shape. When considering a child's face there are a number of curves which outline the important features of the face. Describing these feature curves gives a large amount of information on the shape of the face. This thesis looks to define the shape of children's faces using functions of bending, called curvature functions, of important feature curves. These curvature functions are not only of use to define an object, they are apt for use in the comparison of two or more objects. Methods to produce curvature functions which provide an accurate description of the bending of face curves will be introduced in this thesis. Furthermore, methods to compare the facial shape of groups of children will be discussed. These methods will be used to compare the facial shape of children with a cleft lip and/or palate with control children. There is much recent literature in the area of functional regression where a scalar response can be related to a functional predictor. A novel approach for relating shape to a scalar response using functional regression, with curvature functions as predictors, is discussed and illustrated by a study into the psychological state of ten year old children who were born with a cleft lip or a cleft lip and palate. The aim of this example is to investigate whether any relationship exists between the bending of facial features and the psychological score of the children, and where relationships exist to describe their nature. The thesis consists of four parts. Chapters 1 and 2 introduce the data and give some background to the statistical techniques. Specifically, Chapter 1 briefly introduces the idea of shape and how the shape of objects can be defined using curvature. Furthermore, the two studies into facial shape are introduced which form the basis of the work in this thesis. Chapter 2 gives a broad overview of some standard shape analysis techniques, including Procrustes methods for alignment of objects, and gives further details of methods based on curvature. Functional data analysis techniques which are of use throughout the thesis are also discussed. Part 2 consists of Chapters 3 to 5 which describe methods to find curvature functions that define the shape of important curves on the face and compare these functions to investigate differences between control children and children born with a cleft lip and/or palate. Chapter 3 considers the issues with finding and further analysing the curvature functions of a plane curve whilst Chapter 4 extends the methods to space curves. A method which projects a space curve onto two perpendicular planes and then uses the techniques of Chapter 3 to calculate curvature is introduced to facilitate anatomical interpretation. Whilst the midline profile of a control child is used to illustrate the methods in Chapters 3 and 4, Chapter 5 uses curvature functions to investigate differences between control children and children born with a cleft lip and/or palate in terms of the bending of their upper lips. Part 3 consists of Chapters 6 and 7 which introduce functional regression techniques and use these to investigate potential relationships between the psychological score and facial shape, defined by curvature functions, of cleft children. Methods to both display graphically and formally analyse the regression procedure are discussed in Chapter 6 whilst Chapter 7 uses these methods to provide a systematic analysis of any relationship between psychological score and facial shape. The final part of the thesis presents conclusions discussing both the effectiveness of the methods and some brief anatomical/psychological findings. There are also suggestions of potential future work in the area

Modelling of Orthogonal Craniofacial Profiles

We present a fully-automatic image processing pipeline to build a set of 2D morphable models of three craniofacial profiles from orthogonal viewpoints, side view, front view and top view, using a set of 3D head surface images. Subjects in this dataset wear a close-fitting latex cap to reveal the overall skull shape. Texture-based 3D pose normalization and facial landmarking are applied to extract the profiles from 3D raw scans. Fully-automatic profile annotation, subdivision and registration methods are used to establish dense correspondence among sagittal profiles. The collection of sagittal profiles in dense correspondence are scaled and aligned using Generalised Procrustes Analysis (GPA), before applying principal component analysis to generate a morphable model. Additionally, we propose a new alternative alignment called the Ellipse Centre Nasion (ECN) method. Our model is used in a case study of craniosynostosis intervention outcome evaluation, and the evaluation reveals that the proposed model achieves state-of-the-art results. We make publicly available both the morphable models and the profile dataset used to construct it

Analysis of 3D Face Reconstruction

This thesis investigates the long standing problem of 3D reconstruction from a single 2D face image. Face reconstruction from a single 2D face image is an ill posed problem involving estimation of the intrinsic and the extrinsic camera parameters, light parameters, shape parameters and the texture parameters. The proposed approach has many potential applications in the law enforcement, surveillance, medicine, computer games and the entertainment industries. This problem is addressed using an analysis by synthesis framework by reconstructing a 3D face model from identity photographs. The identity photographs are a widely used medium for face identi cation and can be found on identity cards and passports. The novel contribution of this thesis is a new technique for creating 3D face models from a single 2D face image. The proposed method uses the improved dense 3D correspondence obtained using rigid and non-rigid registration techniques. The existing reconstruction methods use the optical ow method for establishing 3D correspondence. The resulting 3D face database is used to create a statistical shape model. The existing reconstruction algorithms recover shape by optimizing over all the parameters simultaneously. The proposed algorithm simplifies the reconstruction problem by using a step wise approach thus reducing the dimension of the parameter space and simplifying the opti- mization problem. In the alignment step, a generic 3D face is aligned with the given 2D face image by using anatomical landmarks. The texture is then warped onto the 3D model by using the spatial alignment obtained previously. The 3D shape is then recovered by optimizing over the shape parameters while matching a texture mapped model to the target image. There are a number of advantages of this approach. Firstly, it simpli es the optimization requirements and makes the optimization more robust. Second, there is no need to accurately recover the illumination parameters. Thirdly, there is no need for recovering the texture parameters by using a texture synthesis approach. Fourthly, quantitative analysis is used for improving the quality of reconstruction by improving the cost function. Previous methods use qualitative methods such as visual analysis, and face recognition rates for evaluating reconstruction accuracy. The improvement in the performance of the cost function occurs as a result of improvement in the feature space comprising the landmark and intensity features. Previously, the feature space has not been evaluated with respect to reconstruction accuracy thus leading to inaccurate assumptions about its behaviour. The proposed approach simpli es the reconstruction problem by using only identity images, rather than placing eff ort on overcoming the pose, illumination and expression (PIE) variations. This makes sense, as frontal face images under standard illumination conditions are widely available and could be utilized for accurate reconstruction. The reconstructed 3D models with texture can then be used for overcoming the PIE variations