Infrared face recognition: a comprehensive review of methodologies and databases

Automatic face recognition is an area with immense practical potential which includes a wide range of commercial and law enforcement applications. Hence it is unsurprising that it continues to be one of the most active research areas of computer vision. Even after over three decades of intense research, the state-of-the-art in face recognition continues to improve, benefitting from advances in a range of different research fields such as image processing, pattern recognition, computer graphics, and physiology. Systems based on visible spectrum images, the most researched face recognition modality, have reached a significant level of maturity with some practical success. However, they continue to face challenges in the presence of illumination, pose and expression changes, as well as facial disguises, all of which can significantly decrease recognition accuracy. Amongst various approaches which have been proposed in an attempt to overcome these limitations, the use of infrared (IR) imaging has emerged as a particularly promising research direction. This paper presents a comprehensive and timely review of the literature on this subject. Our key contributions are: (i) a summary of the inherent properties of infrared imaging which makes this modality promising in the context of face recognition, (ii) a systematic review of the most influential approaches, with a focus on emerging common trends as well as key differences between alternative methodologies, (iii) a description of the main databases of infrared facial images available to the researcher, and lastly (iv) a discussion of the most promising avenues for future research.Comment: Pattern Recognition, 2014. arXiv admin note: substantial text overlap with arXiv:1306.160

Unifying the Visible and Passive Infrared Bands: Homogeneous and Heterogeneous Multi-Spectral Face Recognition

Face biometrics leverages tools and technology in order to automate the identification of individuals. In most cases, biometric face recognition (FR) can be used for forensic purposes, but there remains the issue related to the integration of technology into the legal system of the court. The biggest challenge with the acceptance of the face as a modality used in court is the reliability of such systems under varying pose, illumination and expression, which has been an active and widely explored area of research over the last few decades (e.g. same-spectrum or homogeneous matching). The heterogeneous FR problem, which deals with matching face images from different sensors, should be examined for the benefit of military and law enforcement applications as well. In this work we are concerned primarily with visible band images (380-750 nm) and the infrared (IR) spectrum, which has become an area of growing interest.;For homogeneous FR systems, we formulate and develop an efficient, semi-automated, direct matching-based FR framework, that is designed to operate efficiently when face data is captured using either visible or passive IR sensors. Thus, it can be applied in both daytime and nighttime environments. First, input face images are geometrically normalized using our pre-processing pipeline prior to feature-extraction. Then, face-based features including wrinkles, veins, as well as edges of facial characteristics, are detected and extracted for each operational band (visible, MWIR, and LWIR). Finally, global and local face-based matching is applied, before fusion is performed at the score level. Although this proposed matcher performs well when same-spectrum FR is performed, regardless of spectrum, a challenge exists when cross-spectral FR matching is performed. The second framework is for the heterogeneous FR problem, and deals with the issue of bridging the gap across the visible and passive infrared (MWIR and LWIR) spectrums. Specifically, we investigate the benefits and limitations of using synthesized visible face images from thermal and vice versa, in cross-spectral face recognition systems when utilizing canonical correlation analysis (CCA) and locally linear embedding (LLE), a manifold learning technique for dimensionality reduction. Finally, by conducting an extensive experimental study we establish that the combination of the proposed synthesis and demographic filtering scheme increases system performance in terms of rank-1 identification rate

Enhanced cyberspace defense with real-time distributed systems using covert channel publish-subscribe broker pattern communications

In this thesis, we propose a novel cyberspace defense solution to the growing sophistication of threats facing networks within the Department of Defense. Current network defense strategies, including traditional intrusion detection and firewall-based perimeter defenses, are ineffective against increasingly sophisticated social engineering attacks such as spear-phishing which exploit individuals with targeted information. These asymmetric attacks are able to bypass current network defense technologies allowing adversaries extended and often unrestricted access to portions of the enterprise. Network defense strategies are hampered by solutions favoring network-centric designs which disregard the security requirements of the specific data and information on the networks. Our solution leverages specific technology characteristics from traditional network defense systems and real-time distributed systems using publish-subscribe broker patterns to form the foundation of a full-spectrum cyber operations capability. Building on this foundation, we present the addition of covert channel communications within the distributed systems framework to protect sensitive Command and Control and Battle Management messaging from adversary intercept and exploitation. Through this combined approach, DoD and Service network defense professionals will be able to meet sophisticated cyberspace threats head-on while simultaneously protecting the data and information critical to warfighting Commands, Services and Agencies.http://archive.org/details/enhancedcyberspa109454049US Air Force (USAF) author.Approved for public release; distribution is unlimited

Target classification in multimodal video

The presented thesis focuses on enhancing scene segmentation and target recognition methodologies via the mobilisation of contextual information. The algorithms developed to achieve this goal utilise multi-modal sensor information collected across varying scenarios, from controlled indoor sequences to challenging rural locations. Sensors are chiefly colour band and long wave infrared (LWIR), enabling persistent surveillance capabilities across all environments. In the drive to develop effectual algorithms towards the outlined goals, key obstacles are identified and examined: the recovery of background scene structure from foreground object ’clutter’, employing contextual foreground knowledge to circumvent training a classifier when labeled data is not readily available, creating a labeled LWIR dataset to train a convolutional neural network (CNN) based object classifier and the viability of spatial context to address long range target classification when big data solutions are not enough. For an environment displaying frequent foreground clutter, such as a busy train station, we propose an algorithm exploiting foreground object presence to segment underlying scene structure that is not often visible. If such a location is outdoors and surveyed by an infra-red (IR) and visible band camera set-up, scene context and contextual knowledge transfer allows reasonable class predictions for thermal signatures within the scene to be determined. Furthermore, a labeled LWIR image corpus is created to train an infrared object classifier, using a CNN approach. The trained network demonstrates effective classification accuracy of 95% over 6 object classes. However, performance is not sustainable for IR targets acquired at long range due to low signal quality and classification accuracy drops. This is addressed by mobilising spatial context to affect network class scores, restoring robust classification capability

Face recognition by means of advanced contributions in machine learning

Face recognition (FR) has been extensively studied, due to both scientific fundamental challenges and current and potential applications where human identification is needed. FR systems have the benefits of their non intrusiveness, low cost of equipments and no useragreement requirements when doing acquisition, among the most important ones. Nevertheless, despite the progress made in last years and the different solutions proposed, FR performance is not yet satisfactory when more demanding conditions are required (different viewpoints, blocked effects, illumination changes, strong lighting states, etc). Particularly, the effect of such non-controlled lighting conditions on face images leads to one of the strongest distortions in facial appearance. This dissertation addresses the problem of FR when dealing with less constrained illumination situations. In order to approach the problem, a new multi-session and multi-spectral face database has been acquired in visible, Near-infrared (NIR) and Thermal infrared (TIR) spectra, under different lighting conditions. A theoretical analysis using information theory to demonstrate the complementarities between different spectral bands have been firstly carried out. The optimal exploitation of the information provided by the set of multispectral images has been subsequently addressed by using multimodal matching score fusion techniques that efficiently synthesize complementary meaningful information among different spectra. Due to peculiarities in thermal images, a specific face segmentation algorithm has been required and developed. In the final proposed system, the Discrete Cosine Transform as dimensionality reduction tool and a fractional distance for matching were used, so that the cost in processing time and memory was significantly reduced. Prior to this classification task, a selection of the relevant frequency bands is proposed in order to optimize the overall system, based on identifying and maximizing independence relations by means of discriminability criteria. The system has been extensively evaluated on the multispectral face database specifically performed for our purpose. On this regard, a new visualization procedure has been suggested in order to combine different bands for establishing valid comparisons and giving statistical information about the significance of the results. This experimental framework has more easily enabled the improvement of robustness against training and testing illumination mismatch. Additionally, focusing problem in thermal spectrum has been also addressed, firstly, for the more general case of the thermal images (or thermograms), and then for the case of facialthermograms from both theoretical and practical point of view. In order to analyze the quality of such facial thermograms degraded by blurring, an appropriate algorithm has been successfully developed. Experimental results strongly support the proposed multispectral facial image fusion, achieving very high performance in several conditions. These results represent a new advance in providing a robust matching across changes in illumination, further inspiring highly accurate FR approaches in practical scenarios.El reconeixement facial (FR) ha estat àmpliament estudiat, degut tant als reptes fonamentals científics que suposa com a les aplicacions actuals i futures on requereix la identificació de les persones. Els sistemes de reconeixement facial tenen els avantatges de ser no intrusius,presentar un baix cost dels equips d’adquisició i no la no necessitat d’autorització per part de l’individu a l’hora de realitzar l'adquisició, entre les més importants. De totes maneres i malgrat els avenços aconseguits en els darrers anys i les diferents solucions proposades, el rendiment del FR encara no resulta satisfactori quan es requereixen condicions més exigents (diferents punts de vista, efectes de bloqueig, canvis en la il·luminació, condicions de llum extremes, etc.). Concretament, l'efecte d'aquestes variacions no controlades en les condicions d'il·luminació sobre les imatges facials condueix a una de les distorsions més accentuades sobre l'aparença facial. Aquesta tesi aborda el problema del FR en condicions d'il·luminació menys restringides. Per tal d'abordar el problema, hem adquirit una nova base de dades de cara multisessió i multiespectral en l'espectre infraroig visible, infraroig proper (NIR) i tèrmic (TIR), sota diferents condicions d'il·luminació. En primer lloc s'ha dut a terme una anàlisi teòrica utilitzant la teoria de la informació per demostrar la complementarietat entre les diferents bandes espectrals objecte d’estudi. L'òptim aprofitament de la informació proporcionada pel conjunt d'imatges multiespectrals s'ha abordat posteriorment mitjançant l'ús de tècniques de fusió de puntuació multimodals, capaces de sintetitzar de manera eficient el conjunt d’informació significativa complementària entre els diferents espectres. A causa de les característiques particulars de les imatges tèrmiques, s’ha requerit del desenvolupament d’un algorisme específic per la segmentació de les mateixes. En el sistema proposat final, s’ha utilitzat com a eina de reducció de la dimensionalitat de les imatges, la Transformada del Cosinus Discreta i una distància fraccional per realitzar les tasques de classificació de manera que el cost en temps de processament i de memòria es va reduir de forma significa. Prèviament a aquesta tasca de classificació, es proposa una selecció de les bandes de freqüències més rellevants, basat en la identificació i la maximització de les relacions d'independència per mitjà de criteris discriminabilitat, per tal d'optimitzar el conjunt del sistema. El sistema ha estat àmpliament avaluat sobre la base de dades de cara multiespectral, desenvolupada pel nostre propòsit. En aquest sentit s'ha suggerit l’ús d’un nou procediment de visualització per combinar diferents bandes per poder establir comparacions vàlides i donar informació estadística sobre el significat dels resultats. Aquest marc experimental ha permès més fàcilment la millora de la robustesa quan les condicions d’il·luminació eren diferents entre els processos d’entrament i test. De forma complementària, s’ha tractat la problemàtica de l’enfocament de les imatges en l'espectre tèrmic, en primer lloc, pel cas general de les imatges tèrmiques (o termogrames) i posteriorment pel cas concret dels termogrames facials, des dels punt de vista tant teòric com pràctic. En aquest sentit i per tal d'analitzar la qualitat d’aquests termogrames facials degradats per efectes de desenfocament, s'ha desenvolupat un últim algorisme. Els resultats experimentals recolzen fermament que la fusió d'imatges facials multiespectrals proposada assoleix un rendiment molt alt en diverses condicions d’il·luminació. Aquests resultats representen un nou avenç en l’aportació de solucions robustes quan es contemplen canvis en la il·luminació, i esperen poder inspirar a futures implementacions de sistemes de reconeixement facial precisos en escenaris no controlats.Postprint (published version

Segmentação de faces em imagens no infravermelho térmico

O objectivo desta dissertação é desenvolver um método para a segmentação automática de faces em imagens capturadas no infravermelho térmico, permitindo uma ampla gama de rotações da face e expressões. A motivação por trás desse esforço é de possibilitar um melhor desempenho dos métodos de reconhecimento de faces em imagens no infravermelho térmico. Ao longo desta dissertação são discutidos oito abordagens diferentes e a comparação dos seus desempenhos com outros três métodos publicados anteriormente. As abordagens propostas são baseadas em modelos estatísticos das intensidades dos pixéis e a aplicação de contornos activos, contudo outras operações de processamento de imagem são realizadas. Estudamos também o desempenho de três abordagens de fusão sob diferentes regras (votação por maioria, operador lógico AND e OR). As experiências foram realizadas num total de 893 imagens de teste de 4 bases de dados públicas disponíveis. Os resultados obtidos melhoram os resultados dos métodos existentes até 31:2% para a primeira medida de erro (E1) e até 39:0% para a segunda medida (E2), dependendo do método e da base de dados. Quanto ao tempo computacional, as nossas propostas podem melhorar até 75:4% quando comparadas com as outras propostas

Face recognition using infrared vision

Au cours de la dernière décennie, la reconnaissance de visage basée sur l’imagerie infrarouge (IR) et en particulier la thermographie IR est devenue une alternative prometteuse aux approches conventionnelles utilisant l’imagerie dans le spectre visible. En effet l’imagerie (visible et infrarouge) trouvent encore des contraintes à leur application efficace dans le monde réel. Bien qu’insensibles à toute variation d’illumination dans le spectre visible, les images IR sont caractérisées par des défis spécifiques qui leur sont propres, notamment la sensibilité aux facteurs qui affectent le rayonnement thermique du visage tels que l’état émotionnel, la température ambiante, la consommation d’alcool, etc. En outre, il est plus laborieux de corriger l’expression du visage et les changements de poses dans les images IR puisque leur contenu est moins riche aux hautes fréquences spatiales ce qui représente en fait une indication importante pour le calage de tout modèle déformable. Dans cette thèse, nous décrivons une nouvelle méthode qui répond à ces défis majeurs. Concrètement, pour remédier aux changements dans les poses et expressions du visage, nous générons une image synthétique frontale du visage qui est canonique et neutre vis-à-vis de toute expression faciale à partir d’une image du visage de pose et expression faciale arbitraires. Ceci est réalisé par l’application d’une déformation affine par morceaux précédée par un calage via un modèle d’apparence active (AAM). Ainsi, une de nos publications est la première publication qui explore l’utilisation d’un AAM sur les images IR thermiques ; nous y proposons une étape de prétraitement qui rehausse la netteté des images thermiques, ce qui rend la convergence de l’AAM rapide et plus précise. Pour surmonter le problème des images IR thermiques par rapport au motif exact du rayonnement thermique du visage, nous le décrivons celui-ci par une représentation s’appuyant sur des caractéristiques anatomiques fiables. Contrairement aux approches existantes, notre représentation n’est pas binaire ; elle met plutôt l’accent sur la fiabilité des caractéristiques extraites. Cela rend la représentation proposée beaucoup plus robuste à la fois à la pose et aux changements possibles de température. L’efficacité de l’approche proposée est démontrée sur la plus grande base de données publique des vidéos IR thermiques des visages. Sur cette base d’images, notre méthode atteint des performances de reconnaissance assez bonnes et surpasse de manière significative les méthodes décrites précédemment dans la littérature. L’approche proposée a également montré de très bonnes performances sur des sous-ensembles de cette base de données que nous avons montée nous-mêmes au sein de notre laboratoire. A notre connaissance, il s’agit de l’une des bases de données les plus importantes disponibles à l’heure actuelle tout en présentant certains défis.Over the course of the last decade, infrared (IR) and particularly thermal IR imaging based face recognition has emerged as a promising complement to conventional, visible spectrum based approaches which continue to struggle when applied in the real world. While inherently insensitive to visible spectrum illumination changes, IR images introduce specific challenges of their own, most notably sensitivity to factors which affect facial heat emission patterns, e.g., emotional state, ambient temperature, etc. In addition, facial expression and pose changes are more difficult to correct in IR images because they are less rich in high frequency details which is an important cue for fitting any deformable model. In this thesis we describe a novel method which addresses these major challenges. Specifically, to normalize for pose and facial expression changes we generate a synthetic frontal image of a face in a canonical, neutral facial expression from an image of the face in an arbitrary pose and facial expression. This is achieved by piecewise affine warping which follows active appearance model (AAM) fitting. This is the first work which explores the use of an AAM on thermal IR images; we propose a pre-processing step which enhances details in thermal images, making AAM convergence faster and more accurate. To overcome the problem of thermal IR image sensitivity to the exact pattern of facial temperature emissions we describe a representation based on reliable anatomical features. In contrast to previous approaches, our representation is not binary; rather, our method accounts for the reliability of the extracted features. This makes the proposed representation much more robust both to pose and scale changes. The effectiveness of the proposed approach is demonstrated on the largest public database of thermal IR images of faces on which it achieves satisfying recognition performance and significantly outperforms previously described methods. The proposed approach has also demonstrated satisfying performance on subsets of the largest video database of the world gathered in our laboratory which will be publicly available free of charge in future. The reader should note that due to the very nature of the feature extraction method in our system (i.e., anatomical based nature of it), we anticipate high robustness of our system to some challenging factors such as the temperature changes. However, we were not able to investigate this in depth due to the limits which exist in gathering realistic databases. Gathering the largest video database considering some challenging factors is one of the other contributions of this research

LiDAR based Biomass Estimation System for Forested Areas

In continental Portugal, forest fires are considered the biggest and most serious cause of forest deterioration and therefore the introduction of forest management mechanisms and biomass monitoring are imperative for a better future. However, conducting field studies on a large scale is a very expensive and time-consuming task. Alternatively, through remote sensing via a LiDAR, it becomes possible to map, with high accuracy, forest parameters such as tree height, diameter at breast height or tree canopy length in order to carry out other relevant estimates such as above ground biomass. In this sense, this dissertation aims to develop a system capable of, through algorithms and filters of point cloud processing, as statistical outlier removal, progressive morphological filters and region growing segmentation, extract in detail,a digital terrain model and correctly detect the number of trees in a given area, proceeding to the measurement of some interesting variables from the point of view of a forest inventory. Thus, testing data of different characteristics, our detection method obtained positive results, with all the average detection rates above 80 %.Em Portugal continental, os incêndios florestais são considerados a maior e mais grave causa de deterioramento da floresta e por isso a introdução de mecanismos de gestão florestal e monitorização da biomassa são imperativos para um futuro melhor. No entanto, realizar estudos de campo em grande escala é uma tarefa muito dispendiosa e demorosa. Em alternativa, através da deteção remota por vias de um LiDAR torna-se possível mapear, com elevado rigor, parâmetros florestais como altura das arvores, diâmetro do tronco ou comprimento da copa da arvore de modo a proceder a outras relevantes estimações como a biomassa. Neste sentido, esta dissertação teve como objetivo o desenvolvimento de um sistema capaz de, através de algoritmos e filtros de processamento de nuvens de pontos, como remoção de outliers estatístico, filtros morfologicos progressivos e segmentação por crescimento de regiões anexas , extrair com detalhe, um modelo digital do terreno e detetar corretamente o número de arvores numa determinada área, procedendo à medição de algumas variáveis interessantes do ponto de vista do inventário florestal. Assim, testando dados de diferentes características, o nosso método de deteção obteve resultados positivos, com todas as taxas deteção média superiores a 80 %