Verificação facial em duas etapas para dispositivos móveis

Orientadores: Jacques Wainer, Fernanda Alcântara AndalóDissertação (mestrado) - Universidade Estadual de Campinas, Instituto de ComputaçãoResumo: Dispositivos móveis, como smartphones e tablets, se tornaram mais populares e acessíveis nos últimos anos. Como consequência de sua ubiquidade, esses aparelhos guardam diversos tipos de informações pessoais (fotos, conversas de texto, coordenadas GPS, dados bancários, entre outros) que só devem ser acessadas pelo dono do dispositivo. Apesar de métodos baseados em conhecimento, como senhas numéricas ou padrões, ainda estejam entre as principais formas de assegurar a identidade do usuário, traços biométricos tem sido utilizados para garantir uma autenticação mais segura e prática. Entre eles, reconhecimento facial ganhou atenção nos últimos anos devido aos recentes avanços nos dispositivos de captura de imagens e na crescente disponibilidade de fotos em redes sociais. Aliado a isso, o aumento de recursos computacionais, com múltiplas CPUs e GPUs, permitiu o desenvolvimento de modelos mais complexos e robustos, como redes neurais profundas. Porém, apesar da evolução das capacidades de dispositivos móveis, os métodos de reconhecimento facial atuais ainda não são desenvolvidos considerando as características do ambiente móvel, como processamento limitado, conectividade instável e consumo de bateria. Neste trabalho, nós propomos um método de verificação facial otimizado para o ambiente móvel. Ele consiste em um procedimento em dois níveis que combina engenharia de características (histograma de gradientes orientados e análise de componentes principais por regiões) e uma rede neural convolucional para verificar se o indivíduo presente em uma imagem corresponde ao dono do dispositivo. Nós também propomos a \emph{Hybrid-Fire Convolutional Neural Network}, uma arquitetura ajustada para dispositivos móveis que processa informação de pares de imagens. Finalmente, nós apresentamos uma técnica para adaptar o limiar de aceitação do método proposto para imagens com características diferentes daquelas presentes no treinamento, utilizando a galeria de imagens do dono do dispositivo. A solução proposta se compara em acurácia aos métodos de reconhecimento facial do estado da arte, além de possuir um modelo 16 vezes menor e 4 vezes mais rápido ao processar uma imagem em smartphones modernos. Por último, nós também organizamos uma base de dados composta por 2873 selfies de 56 identidades capturadas em condições diversas, a qual esperamos que ajude pesquisas futuras realizadas neste cenárioAbstract: Mobile devices, such as smartphones and tablets, had their popularity and affordability greatly increased in recent years. As a consequence of their ubiquity, these devices now carry all sorts of personal data (\emph{e.g.} photos, text conversations, GPS coordinates, banking information) that should be accessed only by the device's owner. Even though knowledge-based procedures, such as entering a PIN or drawing a pattern, are still the main methods to secure the owner's identity, recently biometric traits have been employed for a more secure and effortless authentication. Among them, face recognition has gained more attention in past years due to recent improvements in image-capturing devices and the availability of images in social networks. In addition to that, the increase in computational resources, with multiple CPUs and GPUs, enabled the design of more complex and robust models, such as deep neural networks. Although the capabilities of mobile devices have been growing in past years, most recent face recognition techniques are still not designed considering the mobile environment's characteristics, such as limited processing power, unstable connectivity and battery consumption. In this work, we propose a facial verification method optimized to the mobile environment. It consists of a two-tiered procedure that combines hand-crafted features (histogram of oriented gradients and local region principal component analysis) and a convolutional neural network to verify if the person depicted in a picture corresponds to the device owner. We also propose \emph{Hybrid-Fire Convolutional Neural Network}, an architecture tweaked for mobile devices that process encoded information of a pair of face images. Finally, we expose a technique to adapt our method's acceptance thresholds to images with different characteristics than those present during training, by using the device owner's enrolled gallery. The proposed solution performs a par to the state-of-the-art face recognition methods, while having a model 16 times smaller and 4 times faster when processing an image in recent smartphone models. Finally, we have collected a new dataset of selfie pictures comprising 2873 images from 56 identities with varied capture conditions, that hopefully will support future researches in this scenarioMestradoCiência da ComputaçãoMestre em Ciência da Computaçã

Advances in generative modelling: from component analysis to generative adversarial networks

This Thesis revolves around datasets and algorithms, with a focus on generative modelling. In particular, we first turn our attention to a novel, multi-attribute, 2D facial dataset. We then present deterministic as well as probabilistic Component Analysis (CA) techniques which can be applied to multi-attribute 2D as well as 3D data. We finally present deep learning generative approaches specially designed to manipulate 3D facial data. Most 2D facial datasets that are available in the literature, are: a) automatically or semi-automatically collected and thus contain noisy labels, hindering the benchmarking and comparisons between algorithms. Moreover, they are not annotated for multiple attributes. In the first part of the Thesis, we present the first manually collected and annotated database, which contains labels for multiple attributes. As we demonstrate in a series of experiments, it can be used in a number of applications ranging from image translation to age-invariant face recognition. Moving on, we turn our attention to CA methodologies. CA approaches, although being able to only capture linear relationships between data, can still be proven to be efficient in data such as UV maps or 3D data registered in a common template, since they are well aligned. The introduction of more complex datasets in the literature, which contain labels for multiple attributes, naturally brought the need for novel algorithms that can simultaneously handle multiple attributes. In this Thesis, we cover novel CA approaches which are specifically designed to be utilised in datasets annotated with respect to multiple attributes and can be used in a variety of tasks, such as 2D image denoising and translation, as well as 3D data generation and identification. Nevertheless, while CA methods are indeed efficient when handling registered 3D facial data, linear 3D generative models lack details when it comes to reconstructing or generating finer facial characteristics. To alleviate this, in the final part of this Thesis we propose a novel generative framework harnessing the power of Generative Adversarial Networks.Open Acces

Investigating Social Interactions Using Multi-Modal Nonverbal Features

Every day, humans are involved in social situations and interplays, with the goal of sharing emotions and thoughts, establishing relationships with or acting on other human beings. These interactions are possible thanks to what is called social intelligence, which is the ability to express and recognize social signals produced during the interactions. These signals aid the information exchange and are expressed through verbal and non-verbal behavioral cues, such as facial expressions, gestures, body pose or prosody. Recently, many works have demonstrated that social signals can be captured and analyzed by automatic systems, giving birth to a relatively new research area called social signal processing, which aims at replicating human social intelligence with machines. In this thesis, we explore the use of behavioral cues and computational methods for modeling and understanding social interactions. Concretely, we focus on several behavioral cues in three specic contexts: rst, we analyze the relationship between gaze and leadership in small group interactions. Second, we expand our analysis to face and head gestures in the context of deception detection in dyadic interactions. Finally, we analyze the whole body for group detection in mingling scenarios

Unifying the Visible and Passive Infrared Bands: Homogeneous and Heterogeneous Multi-Spectral Face Recognition

Face biometrics leverages tools and technology in order to automate the identification of individuals. In most cases, biometric face recognition (FR) can be used for forensic purposes, but there remains the issue related to the integration of technology into the legal system of the court. The biggest challenge with the acceptance of the face as a modality used in court is the reliability of such systems under varying pose, illumination and expression, which has been an active and widely explored area of research over the last few decades (e.g. same-spectrum or homogeneous matching). The heterogeneous FR problem, which deals with matching face images from different sensors, should be examined for the benefit of military and law enforcement applications as well. In this work we are concerned primarily with visible band images (380-750 nm) and the infrared (IR) spectrum, which has become an area of growing interest.;For homogeneous FR systems, we formulate and develop an efficient, semi-automated, direct matching-based FR framework, that is designed to operate efficiently when face data is captured using either visible or passive IR sensors. Thus, it can be applied in both daytime and nighttime environments. First, input face images are geometrically normalized using our pre-processing pipeline prior to feature-extraction. Then, face-based features including wrinkles, veins, as well as edges of facial characteristics, are detected and extracted for each operational band (visible, MWIR, and LWIR). Finally, global and local face-based matching is applied, before fusion is performed at the score level. Although this proposed matcher performs well when same-spectrum FR is performed, regardless of spectrum, a challenge exists when cross-spectral FR matching is performed. The second framework is for the heterogeneous FR problem, and deals with the issue of bridging the gap across the visible and passive infrared (MWIR and LWIR) spectrums. Specifically, we investigate the benefits and limitations of using synthesized visible face images from thermal and vice versa, in cross-spectral face recognition systems when utilizing canonical correlation analysis (CCA) and locally linear embedding (LLE), a manifold learning technique for dimensionality reduction. Finally, by conducting an extensive experimental study we establish that the combination of the proposed synthesis and demographic filtering scheme increases system performance in terms of rank-1 identification rate

Personality Recognition For Deception Detection

Personality aims at capturing stable individual characteristics, typically measurable in quantitative terms, that explain and predict observable behavioral differences. Personality has been proved to be very useful in many life outcomes, and there has been huge interests on predicting personality automatically. Previously, there are tremendous amount of approaches successfully predicting personality. However, most previous research on personality detection has used personality scores assigned by annotators based solely on the text or audio clip, and found that predicting self-reported personality is a much more difficult task than predicting observer-report personality. In our study, we will demonstrate how to accurately detect self-reported personality from speech using various technique include feature engineering and machine learning algorithms. Individual speaker differences such as personality play an important role in deception detection, adding considerably to its difficulty. We therefore hypothesize that personality scores may provide useful information to a deception classifier, helping to account for interpersonal differences in verbal and deceptive behavior. In final step of this study, we focus upon the personality differences between deceivers as well as their common characteristics. We helped collect within- and cross-cultural data to train new automatic procedures to identify deceptive behavior in American and Mandarin speakers. We examined whether personality recognition can help to predict individual differences in deceivers’ behavior. Therefore, we embedded personality recognition classifier into the deception classifier using deep neural network to improve the performance of deception detection