Infrared face recognition: a comprehensive review of methodologies and databases

Automatic face recognition is an area with immense practical potential which includes a wide range of commercial and law enforcement applications. Hence it is unsurprising that it continues to be one of the most active research areas of computer vision. Even after over three decades of intense research, the state-of-the-art in face recognition continues to improve, benefitting from advances in a range of different research fields such as image processing, pattern recognition, computer graphics, and physiology. Systems based on visible spectrum images, the most researched face recognition modality, have reached a significant level of maturity with some practical success. However, they continue to face challenges in the presence of illumination, pose and expression changes, as well as facial disguises, all of which can significantly decrease recognition accuracy. Amongst various approaches which have been proposed in an attempt to overcome these limitations, the use of infrared (IR) imaging has emerged as a particularly promising research direction. This paper presents a comprehensive and timely review of the literature on this subject. Our key contributions are: (i) a summary of the inherent properties of infrared imaging which makes this modality promising in the context of face recognition, (ii) a systematic review of the most influential approaches, with a focus on emerging common trends as well as key differences between alternative methodologies, (iii) a description of the main databases of infrared facial images available to the researcher, and lastly (iv) a discussion of the most promising avenues for future research.Comment: Pattern Recognition, 2014. arXiv admin note: substantial text overlap with arXiv:1306.160

Face recognition using infrared vision

Au cours de la dernière décennie, la reconnaissance de visage basée sur l’imagerie infrarouge (IR) et en particulier la thermographie IR est devenue une alternative prometteuse aux approches conventionnelles utilisant l’imagerie dans le spectre visible. En effet l’imagerie (visible et infrarouge) trouvent encore des contraintes à leur application efficace dans le monde réel. Bien qu’insensibles à toute variation d’illumination dans le spectre visible, les images IR sont caractérisées par des défis spécifiques qui leur sont propres, notamment la sensibilité aux facteurs qui affectent le rayonnement thermique du visage tels que l’état émotionnel, la température ambiante, la consommation d’alcool, etc. En outre, il est plus laborieux de corriger l’expression du visage et les changements de poses dans les images IR puisque leur contenu est moins riche aux hautes fréquences spatiales ce qui représente en fait une indication importante pour le calage de tout modèle déformable. Dans cette thèse, nous décrivons une nouvelle méthode qui répond à ces défis majeurs. Concrètement, pour remédier aux changements dans les poses et expressions du visage, nous générons une image synthétique frontale du visage qui est canonique et neutre vis-à-vis de toute expression faciale à partir d’une image du visage de pose et expression faciale arbitraires. Ceci est réalisé par l’application d’une déformation affine par morceaux précédée par un calage via un modèle d’apparence active (AAM). Ainsi, une de nos publications est la première publication qui explore l’utilisation d’un AAM sur les images IR thermiques ; nous y proposons une étape de prétraitement qui rehausse la netteté des images thermiques, ce qui rend la convergence de l’AAM rapide et plus précise. Pour surmonter le problème des images IR thermiques par rapport au motif exact du rayonnement thermique du visage, nous le décrivons celui-ci par une représentation s’appuyant sur des caractéristiques anatomiques fiables. Contrairement aux approches existantes, notre représentation n’est pas binaire ; elle met plutôt l’accent sur la fiabilité des caractéristiques extraites. Cela rend la représentation proposée beaucoup plus robuste à la fois à la pose et aux changements possibles de température. L’efficacité de l’approche proposée est démontrée sur la plus grande base de données publique des vidéos IR thermiques des visages. Sur cette base d’images, notre méthode atteint des performances de reconnaissance assez bonnes et surpasse de manière significative les méthodes décrites précédemment dans la littérature. L’approche proposée a également montré de très bonnes performances sur des sous-ensembles de cette base de données que nous avons montée nous-mêmes au sein de notre laboratoire. A notre connaissance, il s’agit de l’une des bases de données les plus importantes disponibles à l’heure actuelle tout en présentant certains défis.Over the course of the last decade, infrared (IR) and particularly thermal IR imaging based face recognition has emerged as a promising complement to conventional, visible spectrum based approaches which continue to struggle when applied in the real world. While inherently insensitive to visible spectrum illumination changes, IR images introduce specific challenges of their own, most notably sensitivity to factors which affect facial heat emission patterns, e.g., emotional state, ambient temperature, etc. In addition, facial expression and pose changes are more difficult to correct in IR images because they are less rich in high frequency details which is an important cue for fitting any deformable model. In this thesis we describe a novel method which addresses these major challenges. Specifically, to normalize for pose and facial expression changes we generate a synthetic frontal image of a face in a canonical, neutral facial expression from an image of the face in an arbitrary pose and facial expression. This is achieved by piecewise affine warping which follows active appearance model (AAM) fitting. This is the first work which explores the use of an AAM on thermal IR images; we propose a pre-processing step which enhances details in thermal images, making AAM convergence faster and more accurate. To overcome the problem of thermal IR image sensitivity to the exact pattern of facial temperature emissions we describe a representation based on reliable anatomical features. In contrast to previous approaches, our representation is not binary; rather, our method accounts for the reliability of the extracted features. This makes the proposed representation much more robust both to pose and scale changes. The effectiveness of the proposed approach is demonstrated on the largest public database of thermal IR images of faces on which it achieves satisfying recognition performance and significantly outperforms previously described methods. The proposed approach has also demonstrated satisfying performance on subsets of the largest video database of the world gathered in our laboratory which will be publicly available free of charge in future. The reader should note that due to the very nature of the feature extraction method in our system (i.e., anatomical based nature of it), we anticipate high robustness of our system to some challenging factors such as the temperature changes. However, we were not able to investigate this in depth due to the limits which exist in gathering realistic databases. Gathering the largest video database considering some challenging factors is one of the other contributions of this research

A novel multispectral and 2.5D/3D image fusion camera system for enhanced face recognition

The fusion of images from the visible and long-wave infrared (thermal) portions of the spectrum produces images that have improved face recognition performance under varying lighting conditions. This is because long-wave infrared images are the result of emitted, rather than reflected, light and are therefore less sensitive to changes in ambient light. Similarly, 3D and 2.5D images have also improved face recognition under varying pose and lighting. The opacity of glass to long-wave infrared light, however, means that the presence of eyeglasses in a face image reduces the recognition performance. This thesis presents the design and performance evaluation of a novel camera system which is capable of capturing spatially registered visible, near-infrared, long-wave infrared and 2.5D depth video images via a common optical path requiring no spatial registration between sensors beyond scaling for differences in sensor sizes. Experiments using a range of established face recognition methods and multi-class SVM classifiers show that the fused output from our camera system not only outperforms the single modality images for face recognition, but that the adaptive fusion methods used produce consistent increases in recognition accuracy under varying pose, lighting and with the presence of eyeglasses

Multi-Modal Ocular Recognition in presence of occlusion in Mobile Devices

Title from PDF of title page viewed September 18, 2019Dissertation advisor: Reza DerakhshaniVitaIncludes bibliographical references (pages 128-144)Thesis (Ph.D.)--School of Computing and Engineering. University of Missouri--Kansas City, 2018The existence eyeglasses in human faces cause real challenges for ocular, facial, and soft-based (such as eyebrows) biometric recognition due to glasses reflection, shadow, and frame occlusion. In this regard, two operations (eyeglasses detection and eyeglasses segmentation) have been proposed to mitigate the effect of occlusion using eyeglasses. Eyeglasses detection is an important initial step towards eyeglass segmentation. Three schemes of eye glasses detection have been proposed which are non-learning-based, learning-based, and deep learning-based schemes. The non-learning scheme of eyeglasses detection which consists of cascaded filters achieved an overall accuracy of 99.0% for VI SOB and 97.9% for FERET datasets. The learning-based scheme of eyeglass detection consisting of extracting Local Binary Pattern (LBP), Histogram of Gradients (HOG) and fusing them together, then applying classifiers (such as Support Vector Machine (SVM), Multi-Layer Perceptron (MLP), and Linear Discriminant Analysis (LDA)), and fusing the output of these classifiers. The latter obtained a best overall accuracy of about 99.3% on FERET and 100% on VISOB dataset. Besides, the deep learning-based scheme of eye glasses detection showed a comparative study for eyeglasses frame detection using different Convolutional Neural Network (CNN) structures that are applied to Frame Bridge region and extended ocular region. The best CNN model obtained an overall accuracy of 99.96% for ROI consisting of Frame Bridge. Moreover, two schemes of eyeglasses segmentation have been introduced. The first segmentation scheme was cascaded convolutional Neural Network (CNN). This scheme consists of cascaded CNN’s for eyeglasses detection, weight generation, and glasses segmentation, followed by mathematical and binarization operations. The scheme showed a 100% eyeglasses detection and 91% segmentation accuracy by our proposed approach. Also, the second segmentation scheme was the convolutional de-convolutional network. This CNN model has been implemented with main convolutional layers, de-convolutional layers, and one custom (lamda) layer. This scheme achieved better segmentation results of 97% segmentation accuracy over the cascaded approach. Furthermore, two soft biometric re-identification schemes have been introduced with eyeglasses mitigation. The first scheme was eyebrows-based user authentication consists of local, global, deep feature extraction with learning-based matching. The best result of 0.63% EER using score level fusion of handcraft descriptors (HOG, and GIST) with the deep VGG16 descriptor for eyebrow-based user authentication. The second scheme was eyeglass-based user authentication which consisting of eyeglasses segmentation, morphological cleanup, features extraction, and learning-based matching. The best result of 3.44% EER using score level fusion of handcraft descriptors (HOG, and GIST) with the deep VGG16 descriptor for eyeglasses-based user authentication. Also, an EER enhancement of 2.51% for indoor vs. outdoor (In: Out) light set tings was achieved for eyebrow-based authentication after eyeglasses segmentation and removal using Convolutional-Deconvolutional approach followed by in-painting.Introduction -- Background in machine learning and computer vision -- Eyeglasses detection and segmentation -- User authentication using soft-biometric -- Conclusion and future work -- Appendi

Development of a Videographic Pupillometer System for Non-Invasive, Distant Analysis

The goal of this project was to develop a remote system capable of obtaining measurements of pupillary diameter of a subject from a distance of two meters. The system was realized by acquiring images using various optical techniques. The analog image data is processed and returns the current pupil diameter and percentage change

Segmentação de faces em imagens no infravermelho térmico

O objectivo desta dissertação é desenvolver um método para a segmentação automática de faces em imagens capturadas no infravermelho térmico, permitindo uma ampla gama de rotações da face e expressões. A motivação por trás desse esforço é de possibilitar um melhor desempenho dos métodos de reconhecimento de faces em imagens no infravermelho térmico. Ao longo desta dissertação são discutidos oito abordagens diferentes e a comparação dos seus desempenhos com outros três métodos publicados anteriormente. As abordagens propostas são baseadas em modelos estatísticos das intensidades dos pixéis e a aplicação de contornos activos, contudo outras operações de processamento de imagem são realizadas. Estudamos também o desempenho de três abordagens de fusão sob diferentes regras (votação por maioria, operador lógico AND e OR). As experiências foram realizadas num total de 893 imagens de teste de 4 bases de dados públicas disponíveis. Os resultados obtidos melhoram os resultados dos métodos existentes até 31:2% para a primeira medida de erro (E1) e até 39:0% para a segunda medida (E2), dependendo do método e da base de dados. Quanto ao tempo computacional, as nossas propostas podem melhorar até 75:4% quando comparadas com as outras propostas