Methods for multi-spectral image fusion: identifying stable and repeatable information across the visible and infrared spectra

Fusion of images captured from different viewpoints is a well-known challenge in computer vision with many established approaches and applications; however, if the observations are captured by sensors also separated by wavelength, this challenge is compounded significantly. This dissertation presents an investigation into the fusion of visible and thermal image information from two front-facing sensors mounted side-by-side. The primary focus of this work is the development of methods that enable us to map and overlay multi-spectral information; the goal is to establish a combined image in which each pixel contains both colour and thermal information. Pixel-level fusion of these distinct modalities is approached using computational stereo methods; the focus is on the viewpoint alignment and correspondence search/matching stages of processing. Frequency domain analysis is performed using a method called phase congruency. An extensive investigation of this method is carried out with two major objectives: to identify predictable relationships between the elements extracted from each modality, and to establish a stable representation of the common information captured by both sensors. Phase congruency is shown to be a stable edge detector and repeatable spatial similarity measure for multi-spectral information; this result forms the basis for the methods developed in the subsequent chapters of this work. The feasibility of automatic alignment with sparse feature-correspondence methods is investigated. It is found that conventional methods fail to match inter-spectrum correspondences, motivating the development of an edge orientation histogram (EOH) descriptor which incorporates elements of the phase congruency process. A cost function, which incorporates the outputs of the phase congruency process and the mutual information similarity measure, is developed for computational stereo correspondence matching. An evaluation of the proposed cost function shows it to be an effective similarity measure for multi-spectral information

Towards better performance: phase congruency based face recognition

Phase congruency is an edge detector and measurement of the significant feature in the image. It is a robust method against contrast and illumination variation. In this paper, two novel techniques are introduced for developing alow-cost human identification system based on face recognition. Firstly, the valuable phase congruency features, the gradient-edges and their associate dangles are utilized separately for classifying 130 subjects taken from three face databases with the motivation of eliminating the feature extraction phase. By doing this, the complexity can be significantly reduced. Secondly, the training process is modified when a new technique, called averaging-vectors is developed to accelerate the training process and minimizes the matching time to the lowest value. However, for more comparison and accurate evaluation,three competitive classifiers:  Euclidean distance (ED),cosine distance (CD), and Manhattan distance (MD) are considered in this work. The system performance is very competitive and acceptable, where the experimental  results show promising recognition rates with a reasonable matching time

Pedestrian detection in far infrared images

Detection of people in images is a relatively new field of research, but has been widely accepted. The applications are multiple, such as self-labeling of large databases, security systems and pedestrian detection in intelligent transportation systems. Within the latter, the purpose of a pedestrian detector from a moving vehicle is to detect the presence of people in the path of the vehicle. The ultimate goal is to avoid a collision between the two. This thesis is framed with the advanced driver assistance systems, passive safety systems that warn the driver of conditions that may be adverse. An advanced driving assistance system module, aimed to warn the driver about the presence of pedestrians, using computer vision in thermal images, is presented in this thesis. Such sensors are particularly useful under conditions of low illumination.The document is divided following the usual parts of a pedestrian detection system: development of descriptors that define the appearance of people in these kind of images, the application of these descriptors to full-sized images and temporal tracking of pedestrians found. As part of the work developed in this thesis, database of pedestrians in the far infrared spectrum is presented. This database has been used in developing an evaluation of pedestrian detection systems as well as for the development of new descriptors. These descriptors use techniques for the systematic description of the shape of the pedestrian as well as methods to achieve invariance to contrast, illumination or ambient temperature. The descriptors are analyzed and modified to improve their performance in a detection problem, where potential candidates are searched for in full size images. Finally, a method for tracking the detected pedestrians is proposed to reduce the number of miss-detections that occurred at earlier stages of the algorithm. --La detección de personas en imágenes es un campo de investigación relativamente nuevo, pero que ha tenido una amplia acogida. Las aplicaciones son múltiples, tales como auto-etiquetado de grandes bases de datos, sistemas de seguridad y detección de peatones en sistemas inteligentes de transporte. Dentro de este último, la detección de peatones desde un vehículo móvil tiene como objetivo detectar la presencia de personas en la trayectoria del vehículo. EL fin último es evitar una colisión entre ambos. Esta tesis se enmarca en los sistemas avanzados de ayuda a la conducción; sistemas de seguridad pasivos, que advierten al conductor de condiciones que pueden ser adversas. En esta tesis se presenta un módulo de ayuda a la conducción destinado a advertir de la presencia de peatones, mediante el uso de visión por computador en imágenes térmicas. Este tipo de sensores resultan especialmente útiles en condiciones de baja iluminación. El documento se divide siguiendo las partes habituales de una sistema de detección de peatones: desarrollo de descriptores que defina la apariencia de las personas en este tipo de imágenes, la aplicación de estos en imágenes de tamano completo y el seguimiento temporal de los peatones encontrados. Como parte del trabajo desarrollado en esta tesis se presenta una base de datos de peatones en el espectro infrarrojo lejano. Esta base de datos ha sido utilizada para desarrollar una evaluación de sistemas de detección de peatones, así como para el desarrollo de nuevos descriptores. Estos integran técnicas para la descripción sistemática de la forma del peatón, así como métodos para la invariancia al contraste, la iluminación o la temperatura externa. Los descriptores son analizados y modificados para mejorar su rendimiento en un problema de detección, donde se buscan posibles candidatos en una imagen de tamano completo. Finalmente, se propone una método de seguimiento de los peatones detectados para reducir el número de fallos que se hayan producido etapas anteriores del algoritmo

Homogeneous and Heterogeneous Face Recognition: Enhancing, Encoding and Matching for Practical Applications

Face Recognition is the automatic processing of face images with the purpose to recognize individuals. Recognition task becomes especially challenging in surveillance applications, where images are acquired from a long range in the presence of difficult environments. Short Wave Infrared (SWIR) is an emerging imaging modality that is able to produce clear long range images in difficult environments or during night time. Despite the benefits of the SWIR technology, matching SWIR images against a gallery of visible images presents a challenge, since the photometric properties of the images in the two spectral bands are highly distinct.;In this dissertation, we describe a cross spectral matching method that encodes magnitude and phase of multi-spectral face images filtered with a bank of Gabor filters. The magnitude of filtered images is encoded with Simplified Weber Local Descriptor (SWLD) and Local Binary Pattern (LBP) operators. The phase is encoded with Generalized Local Binary Pattern (GLBP) operator. Encoded multi-spectral images are mapped into a histogram representation and cross matched by applying symmetric Kullback-Leibler distance. Performance of the developed algorithm is demonstrated on TINDERS database that contains long range SWIR and color images acquired at a distance of 2, 50, and 106 meters.;Apart from long acquisition range, other variations and distortions such as pose variation, motion and out of focus blur, and uneven illumination may be observed in multispectral face images. Recognition performance of the face recognition matcher can be greatly affected by these distortions. It is important, therefore, to ensure that matching is performed on high quality images. Poor quality images have to be either enhanced or discarded. This dissertation addresses the problem of selecting good quality samples.;The last chapters of the dissertation suggest a number of modifications applied to the cross spectral matching algorithm for matching low resolution color images in near-real time. We show that the method that encodes the magnitude of Gabor filtered images with the SWLD operator guarantees high recognition rates. The modified method (Gabor-SWLD) is adopted in a camera network set up where cameras acquire several views of the same individual. The designed algorithm and software are fully automated and optimized to perform recognition in near-real time. We evaluate the recognition performance and the processing time of the method on a small dataset collected at WVU

Biometric Systems

Because of the accelerating progress in biometrics research and the latest nation-state threats to security, this book's publication is not only timely but also much needed. This volume contains seventeen peer-reviewed chapters reporting the state of the art in biometrics research: security issues, signature verification, fingerprint identification, wrist vascular biometrics, ear detection, face detection and identification (including a new survey of face recognition), person re-identification, electrocardiogram (ECT) recognition, and several multi-modal systems. This book will be a valuable resource for graduate students, engineers, and researchers interested in understanding and investigating this important field of study

Unconstrained Iris Recognition

This research focuses on iris recognition, the most accurate form of biometric identification. The robustness of iris recognition comes from the unique characteristics of the human, and the permanency of the iris texture as it is stable over human life, and the environmental effects cannot easily alter its shape. In most iris recognition systems, ideal image acquisition conditions are assumed. These conditions include a near infrared (NIR) light source to reveal the clear iris texture as well as look and stare constraints and close distance from the capturing device. However, the recognition accuracy of the-state-of-the-art systems decreases significantly when these constraints are relaxed. Recent advances have proposed different methods to process iris images captured in unconstrained environments. While these methods improve the accuracy of the original iris recognition system, they still have segmentation and feature selection problems, which results in high FRR (False Rejection Rate) and FAR (False Acceptance Rate) or in recognition failure. In the first part of this thesis, a novel segmentation algorithm for detecting the limbus and pupillary boundaries of human iris images with a quality assessment process is proposed. The algorithm first searches over the HSV colour space to detect the local maxima sclera region as it is the most easily distinguishable part of the human eye. The parameters from this stage are then used for eye area detection, upper/lower eyelid isolation and for rotation angle correction. The second step is the iris image quality assessment process, as the iris images captured under unconstrained conditions have heterogeneous characteristics. In addition, the probability of getting a mis-segmented sclera portion around the outer ring of the iris is very high, especially in the presence of reflection caused by a visible wavelength light source. Therefore, quality assessment procedures are applied for the classification of images from the first step into seven different categories based on the average of their RGB colour intensity. An appropriate filter is applied based on the detected quality. In the third step, a binarization process is applied to the detected eye portion from the first step for detecting the iris outer ring based on a threshold value defined on the basis of image quality from the second step. Finally, for the pupil area segmentation, the method searches over the HSV colour space for local minima pixels, as the pupil contains the darkest pixels in the human eye. In the second part, a novel discriminating feature extraction and selection based on the Curvelet transform are introduced. Most of the state-of-the-art iris recognition systems use the textural features extracted from the iris images. While these fine tiny features are very robust when extracted from high resolution clear images captured at very close distances, they show major weaknesses when extracted from degraded images captured over long distances. The use of the Curvelet transform to extract 2D geometrical features (curves and edges) from the degraded iris images addresses the weakness of 1D texture features extracted by the classical methods based on textural analysis wavelet transform. Our experiments show significant improvements in the segmentation and recognition accuracy when compared to the-state-of-the-art results

Robust multispectral image-based localisation solutions for autonomous systems

With the recent increase of interest in multispectral imaging, new image-based localisation solutions have emerged. However, its application to visual odometry remains overlooked. Most localisation techniques are still being developed with visible cameras only, because the portability they can offer and the wide variety of cameras available. Yet, other modalities have great potentials for navigation purposes. Infrared imaging for example, provides different information about the scene and is already used to enhance visible images. This is especially the case of far-infrared cameras which can produce images at night and see hot objects like other cars, animals or pedestrians. Therefore, the aim of this thesis is to tackle the lack of research in multispectral localisation and to explore new ways of performing visual odometry accurately with visible and thermal images. First, a new calibration pattern made of LED lights is presented in Chapter 3. Emitting both visible and thermal radiations, it can easily be seen by infrared and visible cameras. Due to its peculiar shape, the whole pattern can be moved around the cameras and automatically identified in the different images recorded. Monocular and stereo calibration are then performed to precisely estimate the camera parameters. Then, a multispectral monocular visual odometry algorithm is proposed in Chapter 4. This generic technique is able to operate in infrared and visible modalities, regardless of the nature of the images. Incoming images are processed at a high frame rate based on a 2D-to-2D unscaled motion estimation method. However, specific keyframes are carefully selected to avoid degenerate cases and a bundle adjustment optimisation is performed on a sliding window to refine the initial estimation. The advantage of visible-thermal odometry is shown on a scenario with extreme illumination conditions, where the limitation of each modality is reached. The simultaneous combination of visible and thermal images for visual odometry is also explored. In Chapter 5, two feature matching techniques are presented and tested in a multispectral stereo visual odometry framework. One method matches features between stereo pairs independently while the other estimates unscaled motion first, before matching the features altogether. Even though these techniques require more processing power to overcome the dissimilarities between V multimodal images, they have the benefit of estimating scaled transformations. Finally, the camera pose estimates obtained with multispectral stereo odometry are fused with inertial data to create a robustified localisation solution which is detailed in Chapter 6. The full state of the system is estimated, including position, velocity, orientation and IMU biases. It is shown that multispectral visual odometry can correct drifting IMU measurements effectively. Furthermore, it is demonstrated that such multi-sensors setups can be beneficial in challenging situations where features cannot be extracted or tracked. In that case, inertial data can be integrated to provide a state estimate while visual odometry cannot

Target detection, tracking, and localization using multi-spectral image fusion and RF Doppler differentials

It is critical for defense and security applications to have a high probability of detection and low false alarm rate while operating over a wide variety of conditions. Sensor fusion, which is the the process of combining data from two or more sensors, has been utilized to improve the performance of a system by exploiting the strengths of each sensor. This dissertation presents algorithms to fuse multi-sensor data that improves system performance by increasing detection rates, lowering false alarms, and improving track performance. Furthermore, this dissertation presents a framework for comparing algorithm error for image registration which is a critical pre-processing step for multi-spectral image fusion. First, I present an algorithm to improve detection and tracking performance for moving targets in a cluttered urban environment by fusing foreground maps from multi-spectral imagery. Most research in image fusion consider visible and long-wave infrared bands; I examine these bands along with near infrared and mid-wave infrared. To localize and track a particular target of interest, I present an algorithm to fuse output from the multi-spectral image tracker with a constellation of RF sensors measuring a specific cellular emanation. The fusion algorithm matches the Doppler differential from the RF sensors with the theoretical Doppler Differential of the video tracker output by selecting the sensor pair that minimizes the absolute difference or root-mean-square difference. Finally, a framework to quantify shift-estimation error for both area- and feature-based algorithms is presented. By exploiting synthetically generated visible and long-wave infrared imagery, error metrics are computed and compared for a number of area- and feature-based shift estimation algorithms. A number of key results are presented in this dissertation. The multi-spectral image tracker improves the location accuracy of the algorithm while improving the detection rate and lowering false alarms for most spectral bands. All 12 moving targets were tracked through the video sequence with only one lost track that was later recovered. Targets from the multi-spectral tracking algorithm were correctly associated with their corresponding cellular emanation for all targets at lower measurement uncertainty using the root-mean-square difference while also having a high confidence ratio for selecting the true target from background targets. For the area-based algorithms and the synthetic air-field image pair, the DFT and ECC algorithms produces sub-pixel shift-estimation error in regions such as shadows and high contrast painted line regions. The edge orientation feature descriptors increase the number of sub-field estimates while improving the shift-estimation error compared to the Lowe descriptor

Doctor of Philosophy

dissertationDigital image processing has wide ranging applications in combustion research. The analysis of digital images is used in practically every scale of studying combustion phenomena from the scale of individual atoms to diagnosing and controlling large-scale combustors. Digital image processing is one of the fastest-growing scientific areas in the world today. From being able to reconstruct low-resolution grayscale images from transmitted signals, the capabilities have grown to enabling machines carrying out tasks that would normally require human vision, perception, and reasoning. Certain applications in combustion science benefit greatly from recent advances in image processing. Unfortunately, since the two fields - combustion and image processing research - stand relatively far from each other, the most recent results are often not known well enough in the areas where they may be applied with great benefits. This work aims to improve the accuracy and reliability of certain measurements in combustion science by selecting, adapting, and implementing the appropriate techniques originally developed in the image processing area. A number of specific applications were chosen that cover a wide range of physical scales of combustion phenomena, and specific image processing methodologies were proposed to improve or enable measurements in studying such phenomena. The selected applications include the description and quantification of combustion-derived carbon nanostructure, the three-dimensional optical diagnostics of combusting pulverized-coal particles and the optical flow velocimetry and quantitative radiation imaging of a pilot-scale oxy-coal flame. In the field of the structural analysis of soot, new structural parameters were derived and the extraction and fidelity of existing ones were improved. In the field of pulverized-coal combustion, the developed methodologies allow for studying the detailed mechanisms of particle combustion in three dimensions. At larger scales, the simultaneous measurement of flame velocity, spectral radiation, and pyrometric properties were realized

Biometric iris image segmentation and feature extraction for iris recognition

PhD ThesisThe continued threat to security in our interconnected world today begs for urgent solution. Iris biometric like many other biometric systems provides an alternative solution to this lingering problem. Although, iris recognition have been extensively studied, it is nevertheless, not a fully solved problem which is the factor inhibiting its implementation in real world situations today. There exists three main problems facing the existing iris recognition systems: 1) lack of robustness of the algorithm to handle non-ideal iris images, 2) slow speed of the algorithm and 3) the applicability to the existing systems in real world situation. In this thesis, six novel approaches were derived and implemented to address these current limitation of existing iris recognition systems. A novel fast and accurate segmentation approach based on the combination of graph-cut optimization and active contour model is proposed to define the irregular boundaries of the iris in a hierarchical 2-level approach. In the first hierarchy, the approximate boundary of the pupil/iris is estimated using a method based on Hough’s transform for the pupil and adapted starburst algorithm for the iris. Subsequently, in the second hierarchy, the final irregular boundary of the pupil/iris is refined and segmented using graph-cut based active contour (GCBAC) model proposed in this work. The segmentation is performed in two levels, whereby the pupil is segmented first before the iris. In order to detect and eliminate noise and reflection artefacts which might introduce errors to the algorithm, a preprocessing technique based on adaptive weighted edge detection and high-pass filtering is used to detect reflections on the high intensity areas of the image while exemplar based image inpainting is used to eliminate the reflections. After the segmentation of the iris boundaries, a post-processing operation based on combination of block classification method and statistical prediction approach is used to detect any super-imposed occluding eyelashes/eyeshadows. The normalization of the iris image is achieved though the rubber sheet model. In the second stage, an approach based on construction of complex wavelet filters and rotation of the filters to the direction of the principal texture direction is used for the extraction of important iris information while a modified particle swam optimization (PSO) is used to select the most prominent iris features for iris encoding. Classification of the iriscode is performed using adaptive support vector machines (ASVM). Experimental results demonstrate that the proposed approach achieves accuracy of 98.99% and is computationally about 2 times faster than the best existing approach.Ebonyi State University and Education Task Fund, Nigeri