Modeling and simulation of adaptive multimodal optical sensors for target tracking in the visible to near infrared

This work investigates an integrated aerial remote sensor design approach to address moving target detection and tracking problems within highly cluttered, dynamic ground-based scenes. Sophisticated simulation methodologies and scene phenomenology validations have resulted in advancements in artificial multimodal truth video synthesis. Complex modeling of novel micro-opto-electro-mechanical systems (MOEMS) devices, optical systems, and detector arrays has resulted in a proof of concept for a state-of-the-art imaging spectropolarimeter sensor model that does not suffer from typical multimodal image registration problems. Test methodology developed for this work provides the ability to quantify performance of a target tracking application with varying ground scenery, flight characteristics, or sensor specifications. The culmination of this research is an end-to-end simulated demonstration of multimodal aerial remote sensing and target tracking. Deeply hidden target recognition is shown to be enhanced through the fusing of panchromatic, hyperspectral, and polarimetric image modalities. The Digital Imaging and Remote Sensing Image Generation model was leveraged to synthesize truth spectropolarimetric sensor-reaching radiance image cubes comprised of coregistered Stokes vector bands in the visible to near-infrared. An intricate synthetic urban scene containing numerous moving vehicular targets was imaged from a virtual sensor aboard an aerial platform encircling a stare point. An adaptive sensor model was designed with a superpixel array of MOEMS devices fabricated atop a division of focal plane detector. Degree of linear polarization (DoLP) imagery is acquired by combining three adjacent micropolarizer outputs within each 2x2 superpixel whose respective transmissions vary with wavelength, relative angle of polarization, and wire-grid spacing. A novel micromirror within each superpixel adaptively relays light between a panchromatic imaging channel and a hyperspectral spectrometer channel. All optical and detector sensor effects were radiometrically modeled using MATLAB and optical lens design software. Orthorectification of all sensor outputs yields multimodal pseudonadir observation video at a fixed ground sampled distance across an area of responsibility. A proprietary MATLAB-based target tracker accomplishes change detection between sequential panchromatic or DoLP observation frames, and queries the sensor for hyperspectral pixels to aid in track initialization and maintenance. Image quality, spectral quality, and tracking performance metrics are reported for varying scenario parameters including target occlusions within the scene, declination angle and jitter of the aerial platform, micropolarizer diattenuation, and spectral/spatial resolution of the adaptive sensor outputs. DoLP observations were found to track moving vehicles better than panchromatic observations at high oblique angles when facing the sensor generally toward the sun. Vehicular occlusions due to tree canopies and parallax effects of tall buildings significantly reduced tracking performance as expected. Smaller MOEMS pixel sizes drastically improved track performance, but also generated a significant number of false tracks. Atmospheric haze from urban aerosols eliminated the tracking utility of DoLP observations, while aerial platform jitter without image stabilization eliminated tracking utility in both modalities. Wire-grid micropolarizers with very low VNIR diattenuation were found to still extinguish enough cross-polarized light to successfully distinguish and track moving vehicles from their urban background. Thus, state-of-the-art lithographic techniques to create finer wire-grid spacings that exhibit high VNIR diattenuation may not be required

InSPECtor: an end-to-end design framework for compressive pixelated hyperspectral instruments

Classic designs of hyperspectral instrumentation densely sample the spatial and spectral information of the scene of interest. Data may be compressed after the acquisition. In this paper we introduce a framework for the design of an optimized, micro-patterned snapshot hyperspectral imager that acquires an optimized subset of the spatial and spectral information in the scene. The data is thereby compressed already at the sensor level, but can be restored to the full hyperspectral data cube by the jointly optimized reconstructor. This framework is implemented with TensorFlow and makes use of its automatic differentiation for the joint optimization of the layout of the micro-patterned filter array as well as the reconstructor. We explore the achievable compression ratio for different numbers of filter passbands, number of scanning frames, and filter layouts using data collected by the Hyperscout instrument. We show resulting instrument designs that take snapshot measurements without losing significant information while reducing the data volume, acquisition time, or detector space by a factor of 40 as compared to classic, dense sampling. The joint optimization of a compressive hyperspectral imager design and the accompanying reconstructor provides an avenue to substantially reduce the data volume from hyperspectral imagers.Comment: 23 pages, 12 figures, published in Applied Optic

Spectral modulation for full linear polarimetry

Linear (spectro) polarimetry is usually performed using separate photon flux measurements after spatial or temporal polarization modulation. Such classical polarimeters are limited in sensitivity and accuracy by systematic effects and noise. We describe a spectral modulation principle that is based on encoding the full linear polarization properties of light in its spectrum. Such spectral modulation is obtained with an optical train of an achromatic quarter-wave retarder, an athermal multiple-order retarder, and a polarizer. The emergent spectral modulation is sinusoidal with its amplitude scaling with the degree of linear polarization and its phase scaling with the angle of linear polarization. The large advantage of this passive setup is that all polarization information is, in principle, contained in a single spectral measurement, thereby eliminating all differential effects that potentially create spurious polarization signals. Since the polarization properties are obtained through curve fitting, the susceptibility to noise is relatively low. We provide general design options for a spectral modulator and describe the design of a prototype modulator. Currently, the setup in combination with a dedicated retrieval algorithm can be used to measure linear polarization signals with a relative accuracy of 5%.Comment: accepted for publication in Applied Optic

Analysis of Polarimetric Synthetic Aperture Radar and Passive Visible Light Polarimetric Imaging Data Fusion for Remote Sensing Applications

The recent launch of spaceborne (TerraSAR-X, RADARSAT-2, ALOS-PALSAR, RISAT) and airborne (SIRC, AIRSAR, UAVSAR, PISAR) polarimetric radar sensors, with capability of imaging through day and night in almost all weather conditions, has made polarimetric synthetic aperture radar (PolSAR) image interpretation and analysis an active area of research. PolSAR image classification is sensitive to object orientation and scattering properties. In recent years, significant work has been done in many areas including agriculture, forestry, oceanography, geology, terrain analysis. Visible light passive polarimetric imaging has also emerged as a powerful tool in remote sensing for enhanced information extraction. The intensity image provides information on materials in the scene while polarization measurements capture surface features, roughness, and shading, often uncorrelated with the intensity image. Advantages of visible light polarimetric imaging include high dynamic range of polarimetric signatures and being comparatively straightforward to build and calibrate. This research is about characterization and analysis of the basic scattering mechanisms for information fusion between PolSAR and passive visible light polarimetric imaging. Relationships between these two modes of imaging are established using laboratory measurements and image simulations using the Digital Image and Remote Sensing Image Generation (DIRSIG) tool. A novel low cost laboratory based S-band (2.4GHz) PolSAR instrument is developed that is capable of capturing 4 channel fully polarimetric SAR image data. Simple radar targets are formed and system calibration is performed in terms of radar cross-section. Experimental measurements are done using combination of the PolSAR instrument with visible light polarimetric imager for scenes capturing basic scattering mechanisms for phenomenology studies. The three major scattering mechanisms studied in this research include single, double and multiple bounce. Single bounce occurs from flat surfaces like lakes, rivers, bare soil, and oceans. Double bounce can be observed from two adjacent surfaces where one horizontal flat surface is near a vertical surface such as buildings and other vertical structures. Randomly oriented scatters in homogeneous media produce a multiple bounce scattering effect which occurs in forest canopies and vegetated areas. Relationships between Pauli color components from PolSAR and Degree of Linear Polarization (DOLP) from passive visible light polarimetric imaging are established using real measurements. Results show higher values of the red channel in Pauli color image (|HH-VV|) correspond to high DOLP from double bounce effect. A novel information fusion technique is applied to combine information from the two modes. In this research, it is demonstrated that the Degree of Linear Polarization (DOLP) from passive visible light polarimetric imaging can be used for separation of the classes in terms of scattering mechanisms from the PolSAR data. The separation of these three classes in terms of the scattering mechanisms has its application in the area of land cover classification and anomaly detection. The fusion of information from these particular two modes of imaging, i.e. PolSAR and passive visible light polarimetric imaging, is a largely unexplored area in remote sensing and the main challenge in this research is to identify areas and scenarios where information fusion between the two modes is advantageous for separation of the classes in terms of scattering mechanisms relative to separation achieved with only PolSAR

Multiple-target tracking using spectropolarimetric imagery

Detection and tracking methods are two hot research topics in the field of multiple target tracking. Often change detection and motion tracking are used to detect and track moving vehicles, but in this thesis new approaches are provided to improve these two aspects. In the detection aspect, a combined detection method is presented to improve target detection techniques. The method of combining RX (Reed-Xiaoli) with change detection has demonstrated good performance in highly cluttered, dynamic ground-based scenes. In the tracking aspect, Kalman filter and Global Nearest Neighbor are applied in motion tracking to predict the location and implement data association respectively. Spectral features are extracted for each vehicle to solve the limitation of motion tracking through feature matching. The Bhattacharyya distance is used as a criterion in the feature matching procedure. Our algorithm has been tested using three sets data. One is a set of multispectral polarimetric imagery acquired by the Multispectral Aerial Passive Polarimeter System (MAPPS). Another two data sets are spectropolarimetric imagery generated by the Digital Imaging and Remote Sensing Image Generation tool. The tracking performance is analyzed by calculating performance metrics: track purity and (Multiple Object Tracking Accuracy ) MOTA. For MAPPS data, the average MOTA and track purity of feature-aided tracking increase 1 percent and 9 percent over those of motion-only tracking respectively. For DIRSIG data with trees, the average track purity of feature-aided tracking in without noise case increases 2 percent over that of motion-only tracking. In this work, we have demonstrated the capability of detection and tracking methods applied in a complex environment

Spectral LADAR: Active Range-Resolved Imaging Spectroscopy

Imaging spectroscopy using ambient or thermally generated optical sources is a well developed technique for capturing two dimensional images with high per-pixel spectral resolution. The per-pixel spectral data is often a sufficient sampling of a material's backscatter spectrum to infer chemical properties of the constituent material to aid in substance identification. Separately, conventional LADAR sensors use quasi-monochromatic laser radiation to create three dimensional images of objects at high angular resolution, compared to RADAR. Advances in dispersion engineered photonic crystal fibers in recent years have made high spectral radiance optical supercontinuum sources practical, enabling this study of Spectral LADAR, a continuous polychromatic spectrum augmentation of conventional LADAR. This imaging concept, which combines multi-spectral and 3D sensing at a physical level, is demonstrated with 25 independent and parallel LADAR channels and generates point cloud images with three spatial dimensions and one spectral dimension. The independence of spectral bands is a key characteristic of Spectral LADAR. Each spectral band maintains a separate time waveform record, from which target parameters are estimated. Accordingly, the spectrum computed for each backscatter reflection is independently and unambiguously range unmixed from multiple target reflections that may arise from transmission of a single panchromatic pulse. This dissertation presents the theoretical background of Spectral LADAR, a shortwave infrared laboratory demonstrator system constructed as a proof-of-concept prototype, and the experimental results obtained by the prototype when imaging scenes at stand off ranges of 45 meters. The resultant point cloud voxels are spectrally classified into a number of material categories which enhances object and feature recognition. Experimental results demonstrate the physical level combination of active backscatter spectroscopy and range resolved sensing to produce images with a level of complexity, detail, and accuracy that is not obtainable with data-level registration and fusion of conventional imaging spectroscopy and LADAR. The capabilities of Spectral LADAR are expected to be useful in a range of applications, such as biomedical imaging and agriculture, but particularly when applied as a sensor in unmanned ground vehicle navigation. Applications to autonomous mobile robotics are the principal motivators of this study, and are specifically addressed

Statistical Methods for Polarimetric Imagery

Estimation theory is applied to a physical model of incoherent polarized light to address problems in polarimetric image registration, restoration, and analysis for electro-optical imaging systems. In the image registration case, the Cramer-Rao lower bound on unbiased joint estimates of the registration parameters and the underlying scene is derived, simplified using matrix methods, and used to explain the behavior of multi-channel linear polarimetric imagers. In the image restoration case, a polarimetric maximum likelihood blind deconvolution algorithm is derived and tested using laboratory and simulated imagery. Finally, a principal components analysis is derived for polarization imaging systems. This analysis expands upon existing research by including an allowance for partially polarized and unpolarized light

PRIMA General Observer Science Book

PRIMA (The PRobe for-Infrared Mission for Astrophysics) is a concept for a far-infrared (IR) observatory. PRIMA features a cryogenically cooled 1.8 m diameter telescope and is designed to carry two science instruments enabling ultra-high sensitivity imaging and spectroscopic studies in the 24 to 235 microns wavelength range. The resulting observatory is a powerful survey and discovery machine, with mapping speeds better by 2 - 4 orders of magnitude with respect to its far-IR predecessors. The bulk of the observing time on PRIMA should be made available to the community through a General Observer (GO) program offering 75% of the mission time over 5 years. In March 2023, the international astronomy community was encouraged to prepare authored contributions articulating scientific cases that are enabled by the telescope massive sensitivity advance and broad spectral coverage, and that could be performed within the context of GO program. This document, the PRIMA General Observer Science Book, is the edited collection of the 76 received contributions.Comment: A. Moullet, T. Kataria, D. Lis, S. Unwin, Y. Hasegawa, E. Mills, C. Battersby, A. Roc, M. Meixner are the editors of the PRIMA General Observer Science Book. The book compiles 76 authored contributions. 399 page

Random Transformations Of Optical Fields And Applications

The interaction of optical waves with material systems often results in complex, seemingly random fields. Because the fluctuations of such fields are typically difficult to analyze, they are regarded as noise to be suppressed. Nevertheless, in many cases the fluctuations of the field result from a linear and deterministic, albeit complicated, interaction between the optical field and the scattering system. As a result, linear systems theory (LST) can be used to frame the scattering problem and highlight situations in which useful information can be extracted from the fluctuations of the scattered field. Three fundamental problems can be posed in LST regardless of the nature of the system: one direct and two inverse problems. The direct problem attempts to predict the response of a known system to a known input. The problem may be simple enough to admit analytical solutions as in the case of homogeneous materials, phase and amplitude screens, and weakly scattering materials; or the problem may require the use of numerical techniques. This dissertation will focus on the two inverse problems, namely the determination of either the excitation field or the scattering system. Traditionally, the excitation determination problem has relied on designing optical systems that respond to the property of interest in a simple, easily quantified way. For example, gratings can be used to map wavelength onto direction of propagation while waveplates and polarizers can map polarization properties onto intensity. The primary difficulty with directly applying the concepts of LST to scattering systems iv is that, while the outputs are still combinations of the inputs, they are not ``simple\u27\u27 combinations such as Fourier transforms or spatially dispersed spectral components of the input spectrum. Instead, the scattered field can be thought of as a massive sampling and mixing of the excitation field. This dissertation will show that such complicated sampling functions can be characterized and that the corresponding scattering medium can then be used as an optical device such as a lens, polarimeter, or spectrometer. The second inverse problem, system determination, is often more difficult because the problem itself may be ill-posed. For scattering systems that are dominated by low-order scattering, the statistical properties of the scattered light may serve as a fingerprint for material discrimination; however, in many situations, the statistical properties of the output do not depend on the material properties. Rather than analyzing the scattered field from one realization of the random interaction, several measurement techniques have been developed that attempt to extract information about the material system from modifications of the scattered field in response to changes in either the excitation or the intrinsic dynamics of the medium itself. One such technique is dynamic light scattering. This dissertation includes an extension to this method that allows for a polarimetric measurement of the scattered light using a reference beam with controllable polarization. Another system determination problem relates to imaging the reflectivity of a target that is being randomly illuminated. It will be demonstrated that an approach based on the correlation between the integrated scattered intensity and the corresponding illumination intensity distribution can prove superior to standard imaging microscop