Context Aided Tracking with Adaptive Hyperspectral Imagery

A methodology for the context-aided tracking of ground vehicles in remote airborne imagery is developed in which a background model is inferred from hyperspectral imagery. The materials comprising the background of a scene are remotely identified and lead to this model. Two model formation processes are developed: a manual method, and method that exploits an emerging adaptive, multiple-object-spectrometer instrument. A semi-automated background modeling approach is shown to arrive at a reasonable background model with minimal operator intervention. A novel, adaptive, and autonomous approach uses a new type of adaptive hyperspectral sensor, and converges to a 66% correct background model in 5% the time of the baseline {a 95% reduction in sensor acquisition time. A multiple-hypothesis-tracker is incorporated, which utilizes background statistics to form track costs and associated track maintenance thresholds. The context-aided system is demonstrated in a high- fidelity tracking testbed, and reduces track identity error by 30%

Real-time Aerial Vehicle Detection and Tracking using a Multi-modal Optical Sensor

Vehicle tracking from an aerial platform poses a number of unique challenges including the small number of pixels representing a vehicle, large camera motion, and parallax error. For these reasons, it is accepted to be a more challenging task than traditional object tracking and it is generally tackled through a number of different sensor modalities. Recently, the Wide Area Motion Imagery sensor platform has received reasonable attention as it can provide higher resolution single band imagery in addition to its large area coverage. However, still, richer sensory information is required to persistently track vehicles or more research on the application of WAMI for tracking is required. With the advancements in sensor technology, hyperspectral data acquisition at video frame rates become possible as it can be cruical in identifying objects even in low resolution scenes. For this reason, in this thesis, a multi-modal optical sensor concept is considered to improve tracking in adverse scenes. The Rochester Institute of Technology Multi-object Spectrometer is capable of collecting limited hyperspectral data at desired locations in addition to full-frame single band imagery. By acquiring hyperspectral data quickly, tracking can be achieved at reasonableframe rates which turns out to be crucial in tracking. On the other hand, the relatively high cost of hyperspectral data acquisition and transmission need to be taken into account to design a realistic tracking. By inserting extended data of the pixels of interest we can address or avoid the unique challenges posed by aerial tracking. In this direction, we integrate limited hyperspectral data to improve measurement-to-track association. Also, a hyperspectral data based target detection method is presented to avoid the parallax effect and reduce the clutter density. Finally, the proposed system is evaluated on realistic, synthetic scenarios generated by the Digital Image and Remote Sensing software

Overview of contextual tracking approaches in information fusion

Proceedings of: Geospatial InfoFusion III. 2-3 May 2013 Baltimore, Maryland, United States.Many information fusion solutions work well in the intended scenarios; but the applications, supporting data, and capabilities change over varying contexts. One example is weather data for electro-optical target trackers of which standards have evolved over decades. The operating conditions of: technology changes, sensor/target variations, and the contextual environment can inhibit performance if not included in the initial systems design. In this paper, we seek to define and categorize different types of contextual information. We describe five contextual information categories that support target tracking: (1) domain knowledge from a user to aid the information fusion process through selection, cueing, and analysis, (2) environment-to-hardware processing for sensor management, (3) known distribution of entities for situation/threat assessment, (4) historical traffic behavior for situation awareness patterns of life (POL), and (5) road information for target tracking and identification. Appropriate characterization and representation of contextual information is needed for future high-level information fusion systems design to take advantage of the large data content available for a priori knowledge target tracking algorithm construction, implementation, and application.Publicad

A Linear Combination of Heuristics Approach to Spatial Sampling Hyperspectral Data for Target Tracking

Persistent surveillance of the battlespace results in better battlespace awareness which aids in obtaining air superiority, winning battles, and saving friendly lives. Although hyperspectral imagery (HSI) data has proven useful for discriminating targets, it presents many challenges as a useful tool in persistent surveillance. A new sensor under development has the potential of overcoming these challenges and transforming our persistent surveillance capability by providing HSI data for a limited number of pixels and grayscale video for the remainder. The challenge of exploiting this new sensor is determining where the HSI data in the sensor\u27s field of view will be the most useful. The approach taken is to use a utility function with components of equal dispersion, periodic poling, missed measurements, and predictive probability of association error (PPAE). The relative importance or optimal weighting of the different types of TOI is accomplished by a genetic algorithm using a multi-objective problem formulation. Experiments show using the utility function with equal weighting results in superior target tracking compared to any individual component by itself, and the equal weighting in close to the optimal solution. The new sensor is successfully exploited resulting in improved persistent surveillance

Multiple-target tracking using spectropolarimetric imagery

Detection and tracking methods are two hot research topics in the field of multiple target tracking. Often change detection and motion tracking are used to detect and track moving vehicles, but in this thesis new approaches are provided to improve these two aspects. In the detection aspect, a combined detection method is presented to improve target detection techniques. The method of combining RX (Reed-Xiaoli) with change detection has demonstrated good performance in highly cluttered, dynamic ground-based scenes. In the tracking aspect, Kalman filter and Global Nearest Neighbor are applied in motion tracking to predict the location and implement data association respectively. Spectral features are extracted for each vehicle to solve the limitation of motion tracking through feature matching. The Bhattacharyya distance is used as a criterion in the feature matching procedure. Our algorithm has been tested using three sets data. One is a set of multispectral polarimetric imagery acquired by the Multispectral Aerial Passive Polarimeter System (MAPPS). Another two data sets are spectropolarimetric imagery generated by the Digital Imaging and Remote Sensing Image Generation tool. The tracking performance is analyzed by calculating performance metrics: track purity and (Multiple Object Tracking Accuracy ) MOTA. For MAPPS data, the average MOTA and track purity of feature-aided tracking increase 1 percent and 9 percent over those of motion-only tracking respectively. For DIRSIG data with trees, the average track purity of feature-aided tracking in without noise case increases 2 percent over that of motion-only tracking. In this work, we have demonstrated the capability of detection and tracking methods applied in a complex environment

A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote Sensin

Modeling and simulation of adaptive multimodal optical sensors for target tracking in the visible to near infrared

This work investigates an integrated aerial remote sensor design approach to address moving target detection and tracking problems within highly cluttered, dynamic ground-based scenes. Sophisticated simulation methodologies and scene phenomenology validations have resulted in advancements in artificial multimodal truth video synthesis. Complex modeling of novel micro-opto-electro-mechanical systems (MOEMS) devices, optical systems, and detector arrays has resulted in a proof of concept for a state-of-the-art imaging spectropolarimeter sensor model that does not suffer from typical multimodal image registration problems. Test methodology developed for this work provides the ability to quantify performance of a target tracking application with varying ground scenery, flight characteristics, or sensor specifications. The culmination of this research is an end-to-end simulated demonstration of multimodal aerial remote sensing and target tracking. Deeply hidden target recognition is shown to be enhanced through the fusing of panchromatic, hyperspectral, and polarimetric image modalities. The Digital Imaging and Remote Sensing Image Generation model was leveraged to synthesize truth spectropolarimetric sensor-reaching radiance image cubes comprised of coregistered Stokes vector bands in the visible to near-infrared. An intricate synthetic urban scene containing numerous moving vehicular targets was imaged from a virtual sensor aboard an aerial platform encircling a stare point. An adaptive sensor model was designed with a superpixel array of MOEMS devices fabricated atop a division of focal plane detector. Degree of linear polarization (DoLP) imagery is acquired by combining three adjacent micropolarizer outputs within each 2x2 superpixel whose respective transmissions vary with wavelength, relative angle of polarization, and wire-grid spacing. A novel micromirror within each superpixel adaptively relays light between a panchromatic imaging channel and a hyperspectral spectrometer channel. All optical and detector sensor effects were radiometrically modeled using MATLAB and optical lens design software. Orthorectification of all sensor outputs yields multimodal pseudonadir observation video at a fixed ground sampled distance across an area of responsibility. A proprietary MATLAB-based target tracker accomplishes change detection between sequential panchromatic or DoLP observation frames, and queries the sensor for hyperspectral pixels to aid in track initialization and maintenance. Image quality, spectral quality, and tracking performance metrics are reported for varying scenario parameters including target occlusions within the scene, declination angle and jitter of the aerial platform, micropolarizer diattenuation, and spectral/spatial resolution of the adaptive sensor outputs. DoLP observations were found to track moving vehicles better than panchromatic observations at high oblique angles when facing the sensor generally toward the sun. Vehicular occlusions due to tree canopies and parallax effects of tall buildings significantly reduced tracking performance as expected. Smaller MOEMS pixel sizes drastically improved track performance, but also generated a significant number of false tracks. Atmospheric haze from urban aerosols eliminated the tracking utility of DoLP observations, while aerial platform jitter without image stabilization eliminated tracking utility in both modalities. Wire-grid micropolarizers with very low VNIR diattenuation were found to still extinguish enough cross-polarized light to successfully distinguish and track moving vehicles from their urban background. Thus, state-of-the-art lithographic techniques to create finer wire-grid spacings that exhibit high VNIR diattenuation may not be required

QUEST Hierarchy for Hyperspectral Face Recognition

Face recognition is an attractive biometric due to the ease in which photographs of the human face can be acquired and processed. The non-intrusive ability of many surveillance systems permits face recognition applications to be used in a myriad of environments. Despite decades of impressive research in this area, face recognition still struggles with variations in illumination, pose and expression not to mention the larger challenge of willful circumvention. The integration of supporting contextual information in a fusion hierarchy known as QUalia Exploitation of Sensor Technology (QUEST) is a novel approach for hyperspectral face recognition that results in performance advantages and a robustness not seen in leading face recognition methodologies. This research demonstrates a method for the exploitation of hyperspectral imagery and the intelligent processing of contextual layers of spatial, spectral, and temporal information. This approach illustrates the benefit of integrating spatial and spectral domains of imagery for the automatic extraction and integration of novel soft features (biometric). The establishment of the QUEST methodology for face recognition results in an engineering advantage in both performance and efficiency compared to leading and classical face recognition techniques. An interactive environment for the testing and expansion of this recognition framework is also provided