Predicting personal traits from facial images using convolutional neural networks augmented with facial landmark information

We consider the task of predicting various traits of a person given an image of their face. We estimate both objective traits, such as gender, ethnicity and hair-color; as well as subjective traits, such as the emotion a person expresses or whether he is humorous or attractive. For sizeable experimentation, we contribute a new Face Attributes Dataset (FAD), having roughly 200,000 attribute labels for the above traits, for over 10,000 facial images. Due to the recent surge of research on Deep Convolutional Neural Networks (CNNs), we begin by using a CNN architecture for estimating facial attributes and show that they indeed provide an impressive baseline performance. To further improve performance, we propose a novel approach that incorporates facial landmark information for input images as an additional channel, helping the CNN learn better attribute-specific features so that the landmarks across various training images hold correspondence. We empirically analyse the performance of our method, showing consistent improvement over the baseline across traits.Microsoft Researc

Efficient prediction of trait judgments from faces using deep neural networks

Judgments of people from their faces are often invalid but influence many social decisions (e.g., legal sentencing), making them an important target for automated prediction. Direct training of deep convolutional neural networks (DCNNs) is difficult because of sparse human ratings, but features obtained from DCNNs pre-trained on other classifications (e.g., object recognition) can predict trait judgments within a given face database. However, it remains unknown if this latter approach generalizes across faces, raters, or traits. Here we directly compare three distinct types of face features, and test them across multiple out-of-sample datasets and traits. DCNNs pre-trained on face identification provided features that generalized the best, and models trained to predict a given trait also predicted several other traits. We demonstrate the flexibility, generalizability, and efficiency of using DCNN features to predict human trait judgments from faces, providing an easily scalable framework for automated prediction of human judgment

Deep Learning Architectures for Heterogeneous Face Recognition

Face recognition has been one of the most challenging areas of research in biometrics and computer vision. Many face recognition algorithms are designed to address illumination and pose problems for visible face images. In recent years, there has been significant amount of research in Heterogeneous Face Recognition (HFR). The large modality gap between faces captured in different spectrum as well as lack of training data makes heterogeneous face recognition (HFR) quite a challenging problem. In this work, we present different deep learning frameworks to address the problem of matching non-visible face photos against a gallery of visible faces. Algorithms for thermal-to-visible face recognition can be categorized as cross-spectrum feature-based methods, or cross-spectrum image synthesis methods. In cross-spectrum feature-based face recognition a thermal probe is matched against a gallery of visible faces corresponding to the real-world scenario, in a feature subspace. The second category synthesizes a visible-like image from a thermal image which can then be used by any commercial visible spectrum face recognition system. These methods also beneficial in the sense that the synthesized visible face image can be directly utilized by existing face recognition systems which operate only on the visible face imagery. Therefore, using this approach one can leverage the existing commercial-off-the-shelf (COTS) and government-off-the-shelf (GOTS) solutions. In addition, the synthesized images can be used by human examiners for different purposes. There are some informative traits, such as age, gender, ethnicity, race, and hair color, which are not distinctive enough for the sake of recognition, but still can act as complementary information to other primary information, such as face and fingerprint. These traits, which are known as soft biometrics, can improve recognition algorithms while they are much cheaper and faster to acquire. They can be directly used in a unimodal system for some applications. Usually, soft biometric traits have been utilized jointly with hard biometrics (face photo) for different tasks in the sense that they are considered to be available both during the training and testing phases. In our approaches we look at this problem in a different way. We consider the case when soft biometric information does not exist during the testing phase, and our method can predict them directly in a multi-tasking paradigm. There are situations in which training data might come equipped with additional information that can be modeled as an auxiliary view of the data, and that unfortunately is not available during testing. This is the LUPI scenario. We introduce a novel framework based on deep learning techniques that leverages the auxiliary view to improve the performance of recognition system. We do so by introducing a formulation that is general, in the sense that can be used with any visual classifier. Every use of auxiliary information has been validated extensively using publicly available benchmark datasets, and several new state-of-the-art accuracy performance values have been set. Examples of application domains include visual object recognition from RGB images and from depth data, handwritten digit recognition, and gesture recognition from video. We also design a novel aggregation framework which optimizes the landmark locations directly using only one image without requiring any extra prior which leads to robust alignment given arbitrary face deformations. Three different approaches are employed to generate the manipulated faces and two of them perform the manipulation via the adversarial attacks to fool a face recognizer. This step can decouple from our framework and potentially used to enhance other landmark detectors. Aggregation of the manipulated faces in different branches of proposed method leads to robust landmark detection. Finally we focus on the generative adversarial networks which is a very powerful tool in synthesizing a visible-like images from the non-visible images. The main goal of a generative model is to approximate the true data distribution which is not known. In general, the choice for modeling the density function is challenging. Explicit models have the advantage of explicitly calculating the probability densities. There are two well-known implicit approaches, namely the Generative Adversarial Network (GAN) and Variational AutoEncoder (VAE) which try to model the data distribution implicitly. The VAEs try to maximize the data likelihood lower bound, while a GAN performs a minimax game between two players during its optimization. GANs overlook the explicit data density characteristics which leads to undesirable quantitative evaluations and mode collapse. This causes the generator to create similar looking images with poor diversity of samples. In the last chapter of thesis, we focus to address this issue in GANs framework

Artificial intelligence in dentistry, orthodontics and orthognathic surgery: A literature review

Artificial intelligence is the ability of machines to work like humans. The concept initially began with the advent of mathematical models which gave calculated outputs based on inputs fed into the system. This was later modified with the introduction of various algorithms which can either give output based on overall data analysis or by selection of information within previous data. It is steadily becoming a favoured mode of treatment due to its efficiency and ability to manage complex conditions in all specialities. In dentistry, artificial intelligence has also popularised over the past few decades. They have been found useful for diagnosis in restorative dentistry, oral pathology and oral surgery. In orthodontics, they have been utilised for diagnosis, assessment of treatment needs, cephalometrics, treatment planning and orthognathic surgeries etc. The current literature review was planned to highlight the uses of artificial intelligence in dentistry, specifically in orthodontics and orthognathic surgery

An end-to-end review of gaze estimation and its interactive applications on handheld mobile devices

In recent years we have witnessed an increasing number of interactive systems on handheld mobile devices which utilise gaze as a single or complementary interaction modality. This trend is driven by the enhanced computational power of these devices, higher resolution and capacity of their cameras, and improved gaze estimation accuracy obtained from advanced machine learning techniques, especially in deep learning. As the literature is fast progressing, there is a pressing need to review the state of the art, delineate the boundary, and identify the key research challenges and opportunities in gaze estimation and interaction. This paper aims to serve this purpose by presenting an end-to-end holistic view in this area, from gaze capturing sensors, to gaze estimation workflows, to deep learning techniques, and to gaze interactive applications.PostprintPeer reviewe

Attractive or Aggressive? A Face Recognition and Machine Learning Approach for Estimating Returns to Visual Appearance

A growing literature documents the presence of appearance premia in labor markets. We analyze appearance premia in a high-profile, high-pay setting: head football coaches at bigtime college sports programs. These employees face job tasks involving repeated interpersonal interaction on multiple fronts and also act as the “face” of their program. We estimate the attractiveness of each employee using a neural network approach, a pre-trained Convolutional Neural Network fine tuned for this application. This approach can eliminate biases induced by volunteer evaluators and limited numbers of photos. We also use this approach to estimate the perceived aggressiveness of each employee based on observable facial features. Aggressiveness can be detected from facial characteristics and may be a trait preferred by managers and customers in this market. Results show clear evidence of a salary premium for less attractive employees. No beauty premium exists in this market. We also find evidence of an aggressiveness premium, as well as evidence of higher attendance at games coached by less attractive and more aggressive appearing coaches, supporting customer based preferences for the premia. We also provide a methodological contribution by incorporating face recognition and computer vision analysis to evaluate employee appearance