Unsupervised Face Alignment by Robust Nonrigid Mapping

We propose a novel approach to unsupervised facial im-age alignment. Differently from previous approaches, that are confined to affine transformations on either the entire face or separate patches, we extract a nonrigid mapping be-tween facial images. Based on a regularized face model, we frame unsupervised face alignment into the Lucas-Kanade image registration approach. We propose a robust optimiza-tion scheme to handle appearance variations. The method is fully automatic and can cope with pose variations and ex-pressions, all in an unsupervised manner. Experiments on a large set of images showed that the approach is effective. 1

Fast algorithms for fitting active appearance models to unconstrained images

Fitting algorithms for Active Appearance Models (AAMs) are usually considered to be robust but slow or fast but less able to generalize well to unseen variations. In this paper, we look into AAM fitting algorithms and make the following orthogonal contributions: We present a simple “project-out” optimization framework that unifies and revises the most well-known optimization problems and solutions in AAMs. Based on this framework, we describe robust simultaneous AAM fitting algorithms the complexity of which is not prohibitive for current systems. We then go on one step further and propose a new approximate project-out AAM fitting algorithm which we coin extended project-out inverse compositional (E-POIC). In contrast to current algorithms, E-POIC is both efficient and robust. Next, we describe a part-based AAM employing a translational motion model, which results in superior fitting and convergence properties. We also show that the proposed AAMs, when trained “in-the-wild” using SIFT descriptors, perform surprisingly well even for the case of unseen unconstrained images. Via a number of experiments on unconstrained human and animal face databases, we show that our combined contributions largely bridge the gap between exact and current approximate methods for AAM fitting and perform comparably with state-of-the-art face alignment algorithms

Unsupervised alignment of objects in images

With the advent of computer vision, various applications become interested to apply it to interpret the 3D and 2D scenes. The main core of computer vision is visual object detection which deals with detecting and representing objects in the image. Visual object detection requires to learn a model of each class type (e.g. car, cat) to be capable to detect objects belonging to the same class. Class learning benefits from a method which automatically aligns class examples making learning more straightforward. The objective of this thesis is to further develop the sate-of-the-art feature-based alignment method which rigidly and automatically aligns object class images to a manually selected seed image. We try to compensate the weakness by providing a method to automatically select the best seed from dataset. Our method first extracts features by utilizing dense sampling method and then scale invariant feature transform (SIFT) descriptor is used to find best matches as initial local feature matches. The final alignment is based on spatial scoring procedure where the initial matches are refined to a set of spatially verified matches. The spatial score is used next to calculate similarity scores. We propose an algorithm which operates on spatial and similarity scores and finally selects the best seed. We also investigate the performance of step-wise alignment using minimum spanning tree (MST) and Dijkstra shortest path instead of direct alignment utilizing a single seed. We conduct our experiments using classes of Caltech-101 for which our unsupervised seed selection and step-wise alignment achieve state-of-the-art performance

Estimating correspondences of deformable objects "in-the-wild"

This is the author accepted manuscript. The final version is available from IEEE via the DOI in this recordDuring the past few years we have witnessed the development of many methodologies for building and fitting Statistical Deformable Models (SDMs). The construction of accurate SDMs requires careful annotation of images with regards to a consistent set of landmarks. However, the manual annotation of a large amount of images is a tedious, laborious and expensive procedure. Furthermore, for several deformable objects, e.g. human body, it is difficult to define a consistent set of landmarks, and, thus, it becomes impossible to train humans in order to accurately annotate a collection of images. Nevertheless, for the majority of objects, it is possible to extract the shape by object segmentation or even by shape drawing. In this paper, we show for the first time, to the best of our knowledge, that it is possible to construct SDMs by putting object shapes in dense correspondence. Such SDMs can be built with much less effort for a large battery of objects. Additionally, we show that, by sampling the dense model, a part-based SDM can be learned with its parts being in correspondence. We employ our framework to develop SDMs of human arms and legs, which can be used for the segmentation of the outline of the human body, as well as to provide better and more consistent annotations for body joints.Engineering and Physical Sciences Research Council (EPSRC)TekesEuropean Community Horizon 202

Relation among images: Modelling, optimization and applications

In the last two decades, the increasing popularity of information technology has led to a dramatic increase in the amount of visual data. Many applications are developed by processing, analyzing and understanding such increasing data; and modelling the relation among images is fundamental to success of many of them. Examples include image classification, content-based image retrieval and face recognition. Given signatures of images, there are many ways to depict the relation among them, such as pairwise distance, kernel function and factor analysis. However, existing methods are still insufficient as they suffer from many real factors such as misalignment of images and inefficiency from nonlinearity. This dissertation focuses on improving the relation modelling, its applications and related optimization. In particular, three aspects of relation modelling are addressed: 1. Integrate image alignment into the relation modelling methods, including image classification and factor analysis, to achieve stability in real applications. 2. Model relation when images are on multiple manifolds. 3. Develop nonlinear relation modelling methods, including tapering kernels for sparsification of kernel-based relation models and developing piecewise linear factor analysis to enjoy both the efficiency of linear models and the flexibility of nonlinear ones. We also discuss future directions of relation modelling in the last chapter from both application and methodology aspects

Automatic analysis of facial actions: a survey

As one of the most comprehensive and objective ways to describe facial expressions, the Facial Action Coding System (FACS) has recently received significant attention. Over the past 30 years, extensive research has been conducted by psychologists and neuroscientists on various aspects of facial expression analysis using FACS. Automating FACS coding would make this research faster and more widely applicable, opening up new avenues to understanding how we communicate through facial expressions. Such an automated process can also potentially increase the reliability, precision and temporal resolution of coding. This paper provides a comprehensive survey of research into machine analysis of facial actions. We systematically review all components of such systems: pre-processing, feature extraction and machine coding of facial actions. In addition, the existing FACS-coded facial expression databases are summarised. Finally, challenges that have to be addressed to make automatic facial action analysis applicable in real-life situations are extensively discussed. There are two underlying motivations for us to write this survey paper: the first is to provide an up-to-date review of the existing literature, and the second is to offer some insights into the future of machine recognition of facial actions: what are the challenges and opportunities that researchers in the field face

Video-based face alignment using efficient sparse and low-rank approach.

Wu, King Keung."August 2011."Thesis (M.Phil.)--Chinese University of Hong Kong, 2011.Includes bibliographical references (p. 119-126).Abstracts in English and Chinese.Abstract --- p.iAcknowledgement --- p.vChapter 1 --- Introduction --- p.1Chapter 1.1 --- Overview of Face Alignment Algorithms --- p.1Chapter 1.1.1 --- Objectives --- p.1Chapter 1.1.2 --- Motivation: Photo-realistic Talking Head --- p.2Chapter 1.1.3 --- Existing methods --- p.5Chapter 1.2 --- Contributions --- p.8Chapter 1.3 --- Outline of the Thesis --- p.11Chapter 2 --- Sparse Signal Representation --- p.13Chapter 2.1 --- Introduction --- p.13Chapter 2.2 --- Problem Formulation --- p.15Chapter 2.2.1 --- l0-nonn minimization --- p.15Chapter 2.2.2 --- Uniqueness --- p.16Chapter 2.3 --- Basis Pursuit --- p.18Chapter 2.3.1 --- From l0-norm to l1-norm --- p.19Chapter 2.3.2 --- l0-l1 Equivalence --- p.20Chapter 2.4 --- l1-Regularized Least Squares --- p.21Chapter 2.4.1 --- Noisy case --- p.22Chapter 2.4.2 --- Over-determined systems of linear equations --- p.22Chapter 2.5 --- Summary --- p.24Chapter 3 --- Sparse Corruptions and Principal Component Pursuit --- p.25Chapter 3.1 --- Introduction --- p.25Chapter 3.2 --- Sparse Corruptions --- p.26Chapter 3.2.1 --- Sparse Corruptions and l1-Error --- p.26Chapter 3.2.2 --- l1-Error and Least Absolute Deviations --- p.28Chapter 3.2.3 --- l1-Regularized l1-Error --- p.29Chapter 3.3 --- Robust Principal Component Analysis (RPCA) and Principal Component Pursuit --- p.31Chapter 3.3.1 --- Principal Component Analysis (PCA) and RPCA --- p.31Chapter 3.3.2 --- Principal Component Pursuit --- p.33Chapter 3.4 --- Experiments of Sparse and Low-rank Approach on Surveillance Video --- p.34Chapter 3.4.1 --- Least Squares --- p.35Chapter 3.4.2 --- l1-Regularized Least Squares --- p.35Chapter 3.4.3 --- l1-Error --- p.36Chapter 3.4.4 --- l1-Regularized l1-Error --- p.36Chapter 3.5 --- Summary --- p.37Chapter 4 --- Split Bregman Algorithm for l1-Problem --- p.45Chapter 4.1 --- Introduction --- p.45Chapter 4.2 --- Bregman Distance --- p.46Chapter 4.3 --- Bregman Iteration for Constrained Optimization --- p.47Chapter 4.4 --- Split Bregman Iteration for l1-Regularized Problem --- p.50Chapter 4.4.1 --- Formulation --- p.51Chapter 4.4.2 --- Advantages of Split Bregman Iteration . . --- p.52Chapter 4.5 --- Fast l1 Algorithms --- p.54Chapter 4.5.1 --- l1-Regularized Least Squares --- p.54Chapter 4.5.2 --- l1-Error --- p.55Chapter 4.5.3 --- l1-Regularized l1-Error --- p.57Chapter 4.6 --- Summary --- p.58Chapter 5 --- Face Alignment Using Sparse and Low-rank Decomposition --- p.61Chapter 5.1 --- Robust Alignment by Sparse and Low-rank Decomposition for Linearly Correlated Images (RASL) --- p.61Chapter 5.2 --- Problem Formulation --- p.62Chapter 5.2.1 --- Theory --- p.62Chapter 5.2.2 --- Algorithm --- p.64Chapter 5.3 --- Direct Extension of RASL: Multi-RASL --- p.66Chapter 5.3.1 --- Formulation --- p.66Chapter 5.3.2 --- Algorithm --- p.67Chapter 5.4 --- Matlab Implementation Details --- p.68Chapter 5.4.1 --- Preprocessing --- p.70Chapter 5.4.2 --- Transformation --- p.73Chapter 5.4.3 --- Jacobian Ji --- p.74Chapter 5.5 --- Experiments --- p.75Chapter 5.5.1 --- Qualitative Evaluations Using Small Dataset --- p.76Chapter 5.5.2 --- Large Dataset Test --- p.81Chapter 5.5.3 --- Conclusion --- p.85Chapter 5.6 --- Sensitivity analysis on selection of references --- p.87Chapter 5.6.1 --- References from consecutive frames --- p.88Chapter 5.6.2 --- References from RASL-aligned images --- p.91Chapter 5.7 --- Summary --- p.92Chapter 6 --- Extension of RASL for video: One-by-One Approach --- p.96Chapter 6.1 --- One-by-One Approach --- p.96Chapter 6.1.1 --- Motivation --- p.97Chapter 6.1.2 --- Algorithm --- p.97Chapter 6.2 --- Choices of Optimization --- p.101Chapter 6.2.1 --- l1-Regularized Least Squares --- p.101Chapter 6.2.2 --- l1-Error --- p.102Chapter 6.2.3 --- l1-Regularized l1-Error --- p.103Chapter 6.3 --- Experiments --- p.104Chapter 6.3.1 --- Evaluation for Different l1 Algorithms --- p.104Chapter 6.3.2 --- Conclusion --- p.108Chapter 6.4 --- Exploiting Property of Video --- p.109Chapter 6.5 --- Summary --- p.110Chapter 7 --- Conclusion and Future Work --- p.112Chapter A --- Appendix --- p.117Bibliography --- p.11