An Analysis of multimodal sensor fusion for target detection in an urban environment

This work makes a compelling case for simulation as an attractive tool in designing cutting-edge remote sensing systems to generate the sheer volume of data required for a reasonable trade study. The generalized approach presented here allows multimodal system designers to tailor target and sensor parameters for their particular scenarios of interest via synthetic image generation tools, ensuring that resources are best allocated while sensors are still in the design phase. Additionally, sensor operators can use the customizable process showcased here to optimize image collection parameters for existing sensors. In the remote sensing community, polarimetric capabilities are often seen as a tool without a widely accepted mission. This study proposes incorporating a polarimetric and spectral sensor in a multimodal architecture to improve target detection performance in an urban environment. Two novel multimodal fusion algorithms are proposed--one for the pixel level, and another for the decision level. A synthetic urban scene is rendered for 355 unique combinations of illumination condition and sensor viewing geometry with the Digital Imaging and Remote Sensing Image Generation (DIRSIG) model, and then validated to ensure the presence of enough background clutter. The utility of polarimetric information is shown to vary with the sun-target-sensor geometry, and the decision fusion algorithm is shown to generally outperform the pixel fusion algorithm. The results essentially suggest that polarimetric information may be leveraged to restore the capabilities of a spectral sensor if forced to image under less than ideal circumstances

Modeling and simulation of adaptive multimodal optical sensors for target tracking in the visible to near infrared

This work investigates an integrated aerial remote sensor design approach to address moving target detection and tracking problems within highly cluttered, dynamic ground-based scenes. Sophisticated simulation methodologies and scene phenomenology validations have resulted in advancements in artificial multimodal truth video synthesis. Complex modeling of novel micro-opto-electro-mechanical systems (MOEMS) devices, optical systems, and detector arrays has resulted in a proof of concept for a state-of-the-art imaging spectropolarimeter sensor model that does not suffer from typical multimodal image registration problems. Test methodology developed for this work provides the ability to quantify performance of a target tracking application with varying ground scenery, flight characteristics, or sensor specifications. The culmination of this research is an end-to-end simulated demonstration of multimodal aerial remote sensing and target tracking. Deeply hidden target recognition is shown to be enhanced through the fusing of panchromatic, hyperspectral, and polarimetric image modalities. The Digital Imaging and Remote Sensing Image Generation model was leveraged to synthesize truth spectropolarimetric sensor-reaching radiance image cubes comprised of coregistered Stokes vector bands in the visible to near-infrared. An intricate synthetic urban scene containing numerous moving vehicular targets was imaged from a virtual sensor aboard an aerial platform encircling a stare point. An adaptive sensor model was designed with a superpixel array of MOEMS devices fabricated atop a division of focal plane detector. Degree of linear polarization (DoLP) imagery is acquired by combining three adjacent micropolarizer outputs within each 2x2 superpixel whose respective transmissions vary with wavelength, relative angle of polarization, and wire-grid spacing. A novel micromirror within each superpixel adaptively relays light between a panchromatic imaging channel and a hyperspectral spectrometer channel. All optical and detector sensor effects were radiometrically modeled using MATLAB and optical lens design software. Orthorectification of all sensor outputs yields multimodal pseudonadir observation video at a fixed ground sampled distance across an area of responsibility. A proprietary MATLAB-based target tracker accomplishes change detection between sequential panchromatic or DoLP observation frames, and queries the sensor for hyperspectral pixels to aid in track initialization and maintenance. Image quality, spectral quality, and tracking performance metrics are reported for varying scenario parameters including target occlusions within the scene, declination angle and jitter of the aerial platform, micropolarizer diattenuation, and spectral/spatial resolution of the adaptive sensor outputs. DoLP observations were found to track moving vehicles better than panchromatic observations at high oblique angles when facing the sensor generally toward the sun. Vehicular occlusions due to tree canopies and parallax effects of tall buildings significantly reduced tracking performance as expected. Smaller MOEMS pixel sizes drastically improved track performance, but also generated a significant number of false tracks. Atmospheric haze from urban aerosols eliminated the tracking utility of DoLP observations, while aerial platform jitter without image stabilization eliminated tracking utility in both modalities. Wire-grid micropolarizers with very low VNIR diattenuation were found to still extinguish enough cross-polarized light to successfully distinguish and track moving vehicles from their urban background. Thus, state-of-the-art lithographic techniques to create finer wire-grid spacings that exhibit high VNIR diattenuation may not be required

Polarimetric Calibration and Characterization of the Telops Field Portable Polarimetric-Hyperspectral Imager

The Telops polarimetric-hyperspectral imager combines polarimetric and hyperspectral technologies to enable enhanced scene characterization. The Defense Threat Reduction Agency funded research at AFIT to leverage this capability to provide more accurate scene information to radiation transport models that will allow for more effective location of radiation sources within a region of interest. To support the objectives of the DTRA effort, there is a requirement for highly accurate radiometric, polarimetric, and spectral data on a pixel-by-pixel basis. The complex nature of the Telops instrument combined with working in the thermal IR waveband makes achieving this accuracy a challenge. This thesis develops a calibration methodology that enables high data accuracy in all three domains. In the process, a mathematical calibration framework was developed that links standard Fourier transform spectrometer (FTS) calibration with standard polarimetric calibration in a straightforward manner. This provided a framework for understanding the influence of various instrument parameters (both ideal and non-ideal) on ultimate calibration performance. The framework developed is utilized to quantify the non-idealities of the system and to characterize the performance of the spectro-polarimetric calibration. Additionally, fundamental performance limits are characterized including the noise equivalent spectral radiance and noise equivalent degree of linear polarization of the system

Incorporation of Polarization Into the DIRSIG Synthetic Image Generation Model

The Digital Imaging and Remote Sensing Synthetic Image Generation (DIRSIG) model uses a quantitative first principles approach to generate synthetic hyperspectral imagery. This paper presents the methods used to add modeling of polarization phenomenology. The radiative transfer equations were modified to use Stokes vectors for the radiance values and Mueller matrices for the energy-matter interactions. The use of Stokes vectors enables a full polarimetric characterization of the illumination and sensor reaching radiances. The bi-directional reflectance distribution function (BRDF) module was rewritten and modularized to accommodate a variety of polarized and unpolarized BRDF models. Two new BRDF models based on Torrance- Sparrow and Beard-Maxwell were added to provide polarized BRDF estimations. The sensor polarization characteristics are modeled using Mueller matrix transformations on a per pixel basis. All polarized radiative transfer calculations are performed spectrally to preserve the hyperspectral capabilities of DIRSIG. Integration over sensor bandpasses is handled by the sensor module

Modeling polarimetric imaging using DIRSIG

The remote sensing community is beginning to recognize the potential benefit of exploiting polarimetric signatures. The ability to accurately model polarimetric phenomenology in a remote sensing system will assist efforts in system design, algorithm development, phenomenology studies, and analyst training. This dissertation lays the ground work for enhancing the current Digital Imaging and Remote Sens ing Laboratory\u27s Synthetic Image Generation (DIRSIG) model to include polarimetric phenomenology. The current modeling capabilities are discussed along with the theoretical background required to expand upon the current state of the art. Methods for modeling and estimating polarimetric signatures and phenomenology from start to end in a typical remote sensing system are presented. A series of simple simulations were conducted to assess the performance of the new polarimetric capabilities. Analysis was performed to characterize the individual models and the collected performance of the models

Deep learning in remote sensing: a review

Standing at the paradigm shift towards data-intensive science, machine learning techniques are becoming increasingly important. In particular, as a major breakthrough in the field, deep learning has proven as an extremely powerful tool in many fields. Shall we embrace deep learning as the key to all? Or, should we resist a 'black-box' solution? There are controversial opinions in the remote sensing community. In this article, we analyze the challenges of using deep learning for remote sensing data analysis, review the recent advances, and provide resources to make deep learning in remote sensing ridiculously simple to start with. More importantly, we advocate remote sensing scientists to bring their expertise into deep learning, and use it as an implicit general model to tackle unprecedented large-scale influential challenges, such as climate change and urbanization.Comment: Accepted for publication IEEE Geoscience and Remote Sensing Magazin

Hyperspectral Imaging System Model Implementation and Analysis

In support of hyperspectral imaging system design and parameter trade-off research, an analytical end-to-end model to simulate the remote sensing system pipeline and to forecast remote sensing system performance has been implemented. It is also being made available to the remote sensing community through a website. Users are able to forecast hyperspectral imaging system performance by defining an observational scenario along with imaging system parameters. For system modeling, the implemented analytical model includes scene, sensor and target characteristics as well as atmospheric features, background spectral reflectance statistics, sensor specifications and target class reflectance statistics. The sensor model has been extended to include the airborne ProspecTIR instrument. To validate the analytical model, experiments were designed and conducted. The predictive system model has been verified by comparing the forecast results to ones obtained using real world data collected during the RIT SHARE 2012 collection. Results include the use of large calibration panels to show the predicted radiance consistent with the collected data. Grass radiance predicted from ground truth reflectance data also compare well with the real world collected data, and an eigenvector analysis also supports the validity of the predictions. Two examples of subpixel target detection scenario are presented. One is to detect subpixel wood yellow painted planks in an asphalt playground, and the other is to detect subpixel green painted wood planks in grass. To validate our system performance, the detection performance are analyzed using receiver operating characteristic (ROC) curves in a comprehensive scenario setting. The predicted ROC result of the yellow planks matches well the ROC derived from collected data. However, the predicted ROC curve of green planks differs from collected data ROC curve. Additional experiments were conducted and analyzed to discuss the possible reasons of the mismatch including scene characterization inaccuracy. Several subpixel target detection parameter trade-off analyses are given, including relative calibration error vs SNR, the relationship among probability of detection, meteorological range, pixel fill factor, relative calibration error and false alarm rate. These trade-off analyses explain the utility of this model for hyperspectral imaging system design and research

Polarimetric remote sensing system analysis: Digital Imaging and Remote Sensing Image Generation (DIRSIG) model validation and impact of polarization phenomenology on material discriminability

In addition to spectral information acquired by traditional multi/hyperspectral systems, passive electro optical and infrared (EO/IR) polarimetric sensors also measure the polarization response of different materials in the scene. Such an imaging modality can be useful in improving surface characterization; however, the characteristics of polarimetric systems have not been completely explored by the remote sensing community. Therefore, the main objective of this research was to advance our knowledge in polarimetric remote sensing by investigating the impact of polarization phenomenology on material discriminability. The first part of this research focuses on system validation, where the major goal was to assess the fidelity of the polarimetric images simulated using the Digital Imaging and Remote Sensing Image Generation (DIRSIG) model. A theoretical framework, based on polarization vision models used for animal vision studies and industrial defect detection applications, was developed within which the major components of the polarimetric image chain were validated. In the second part of this research, a polarization physics based approach for improved material discriminability was proposed. This approach utilizes the angular variation in the polarization response to infer the physical characteristics of the observed surface by imaging the scene in three different view directions. The usefulness of the proposed approach in improving detection performance in the absence of apriori knowledge about the target geometry was demonstrated. Sensitivity analysis of the proposed system for different scene related parameters was performed to identify the imaging conditions under which the material discriminability is maximized. Furthermore, the detection performance of the proposed polarimetric system was compared to that of the hyperspectral system to identify scenarios where polarization information can be very useful in improving the target contrast

EAGLE 2006 – Multi-purpose, multi-angle and multi-sensor in-situ and airborne campaigns over grassland and forest

EAGLE2006 - an intensive field campaign - was carried out in the Netherlands from the 8th until the 18th of June 2006. Several airborne sensors - an optical imaging sensor, an imaging microwave radiometer, and a flux airplane – were used and extensive ground measurements were conducted over one grassland (Cabauw) site and two forest sites (Loobos & Speulderbos) in the central part of the Netherlands, in addition to the acquisition of multi-angle and multi-sensor satellite data. The data set is both unique and urgently needed for the development and validation of models and inversion algorithms for quantitative surface parameter estimation and process studies. EAGLE2006 was led by the Department of Water Resources of the International Institute for Geo-Information Science and Earth Observation and originated from the combination of a number of initiatives coming under different funding. The objectives of the EAGLE2006 campaign were closely related to the objectives of other ESA Campaigns (SPARC2004, Sen2Flex2005 and especially AGRISAR2006). However, one important objective of the campaign is to build up a data base for the investigation and validation of the retrieval of bio-geophysical parameters, obtained at different radar frequencies (X-, C- and L-Band) and at hyperspectral optical and thermal bands acquired over vegetated fields (forest and grassland). As such, all activities were related to algorithm development for future satellite missions such as Sentinels and for satellite validations for MERIS, MODIS as well as AATSR and ASTER thermal data validation, with activities also related to the ASAR sensor on board ESA’s Envisat platform and those on EPS/MetOp and SMOS. Most of the activities in the campaign are highly relevant for the EU GEMS EAGLE project, but also issues related to retrieval of biophysical parameters from MERIS and MODIS as well as AATSR and ASTER data were of particular relevance to the NWO-SRON EcoRTM project, while scaling issues and complementary between these (covering only local sites) and global sensors such as MERIS/SEVIRI, EPS/MetOP and SMOS were also key elements for the SMOS cal/val project and the ESA-MOST DRAGON programme. This contribution describes the mission objectives and provides an overview of the airborne and field campaigns