Spectral Target Detection using Physics-Based Modeling and a Manifold Learning Technique

Identification of materials from calibrated radiance data collected by an airborne imaging spectrometer depends strongly on the atmospheric and illumination conditions at the time of collection. This thesis demonstrates a methodology for identifying material spectra using the assumption that each unique material class forms a lower-dimensional manifold (surface) in the higher-dimensional spectral radiance space and that all image spectra reside on, or near, these theoretic manifolds. Using a physical model, a manifold characteristic of the target material exposed to varying illumination and atmospheric conditions is formed. A graph-based model is then applied to the radiance data to capture the intricate structure of each material manifold, followed by the application of the commute time distance (CTD) transformation to separate the target manifold from the background. Detection algorithms are then applied in the CTD subspace. This nonlinear transformation is based on a random walk on a graph and is derived from an eigendecomposition of the pseudoinverse of the graph Laplacian matrix. This work provides a geometric interpretation of the CTD transformation, its algebraic properties, the atmospheric and illumination parameters varied in the physics-based model, and the influence the target manifold samples have on the orientation of the coordinate axes in the transformed space. This thesis concludes by demonstrating improved detection results in the CTD subspace as compared to detection in the original spectral radiance space

Using Remote Sensing to Characterize Disturbance during a Severe Drought in the Sierra Nevada

Between 2012 and 2016, California experienced an extreme period of drought and high temperatures. During this period there were two particularly notable disturbances in the southern Sierra Nevada. In 2013, the first major disturbance occurred in the form of the Rim Fire. At 1,041 km2, the Rim Fire is the largest fire ever recorded in the Sierra Nevada. Throughout the drought, but particularly in the latter years, there was also epidemic levels of tree mortality, particularly mortality tied to native bark beetles (Dendroctonus spp.) killing pines (Pinus spp.).Using the southern Sierra Nevada as a case study, I investigated the novel insights a next-generation imaging spectroscopy satellite would be able to give in understanding fire and tree mortality globally. Specifically, I showed the potential for imaging spectroscopy based spectral mixture analysis (SMA) as an assessment of fire severity. SMA cover fractions allow for a remotely sensed fire severity metric that would be more readily compared at the global level than those currently in use, such as difference normalized burn ratio (dNBR). I also demonstrated that using the random forest machine learning algorithm, a simulated spaceborne imaging spectrometer would be able to more accurately identify the location of red stage tree mortality compared to existing multispectral satellites such as Landsat. In addition, remote sensing was used to gain an understanding of the impact and drivers of tree mortality. For a 2,240 ha watershed, a model of tree crown locations, with species and height identified, was created based on a combination of high spatial resolution airborne imaging spectroscopy and lidar. Then, high-resolution multispectral imagery was interpreted to determine a tree’s 2016 status. In the area investigated, the net effect of the drought was to reduce the number of live conifer stems taller than 15 m at the crown level by 75%, primarily due to the death of ponderosa pine. Finally, the factors that distinguished conifers that were alive in 2016 from conifers that died between 2015 and 2016, both within the 2,240 ha watershed and across the southern Sierra Nevada were examined. Trees that survived were typically associated with being located in stands with tree species and height class heterogeneity. Stands in wetter, cooler parts of the Sierra Nevada during the drought were also more likely to survive

Visible and Near Infrared imaging spectroscopy and the exploration of small scale hydrothermally altered and hydrated environments on Earth and Mars

The use of Visible and Near Infrared (VNIR) imaging spectroscopy is a cornerstone of planetary exploration. This work shall present an investigation into the limitations of scale, both spectral and spatial, in the utility of VNIR images for identifying small scale hydrothermal and potential hydrated environments on Mars, and regions of the Earth that can serve as martian analogues. Such settings represent possible habitable environments; important locations for astrobiological research. The ESA/Roscosmos ExoMars rover PanCam captures spectrally coarse but spatially high resolution VNIR images. This instrument is still in development and the first field trial of an emulator fitted with the final set of geological filters is presented here. Efficient image analysis techniques are explored and the ability to accurately characterise a hydrothermally altered region using PanCam data products is established. The CRISM orbital instrument has been returning hyperspectral VNIR images with an 18 m2 pixel resolution since 2006. The extraction of sub-pixel information from CRISM pixels using Spectral Mixture Analysis (SMA) algorithms is explored. Using synthetic datasets a full SMA pipeline consisting of publically available Matlab algorithms and optimised for investigation of mineralogically complex hydrothermal suites is developed for the first time. This is validated using data from Námafjall in Iceland, the region used to field trial the PanCam prototype. The pipeline is applied to CRISM images covering four regions on Mars identified as having potentially undergone hydrothermal alteration in their past. A second novel use of SMA to extract a unique spectral signature for the potentially hydrated Recurring Slope Lineae features on Mars is presented. The specific methodology presented shows promise and future improvements are suggested. The importance of combining different scales of data and recognising their limitations is discussed based on the results presented and ways in which to take the results presented in this thesis forward are given

Estimating Information in Earth System Data with Machine Learning

El aprendizaje automático ha hecho grandes avances en la ciencia e ingeniería actuales en general y en las ciencias de la Tierra en particular. Sin embargo, los datos de la Tierra plantean problemas particularmente difíciles para el aprendizaje automático debido no sólo al volumen de datos implicado, sino también por la presencia de correlaciones no lineales tanto espaciales como temporales, por una gran diversidad de fuentes de ruido y de incertidumbre, así como por la heterogeneidad de las fuentes de información involucradas. Más datos no implica necesariamente más información. Por lo tanto, extraer conocimiento y contenido informativo mediante el análisis y el modelado de datos resulta crucial, especialmente ahora donde el volumen y la heterogeneidad de los datos aumentan constantemente. Este hecho requiere avances en métodos que puedan cuantificar la información y caracterizar las distribuciones e incertidumbres con precisión. Cuantificar el contenido informativo a los datos y los modelos de nuestro sistema son problemas no resueltos en estadística y el aprendizaje automático. Esta tesis introduce nuevos modelos de aprendizaje automático para extraer conocimiento e información a partir de datos de observación de la Tierra. Proponemos métodos núcleo ('kernel methods'), procesos gaussianos y gaussianización multivariada para tratar la incertidumbre y la cuantificación de la información, y aplicamos estos métodos a una amplia gama de problemas científicos del sistema terrestre. Estos conllevan muchos tipos de problemas de aprendizaje, incluida la clasificación, regresión, estimación de densidad, síntesis, propagación de errores y estimación de medidas teóricas de la información. También demostramos cómo funcionan estos métodos con diferentes fuentes de datos, provenientes de distintos sensores (radar, multiespectrales, hiperespectrales), productos de datos (observaciones, reanálisis y simulaciones de modelos) y cubos de datos (agregados de varias fuentes de datos espacial-temporales ). Las metodologías presentadas nos permiten cuantificar y visualizar cuáles son las características relevantes que gobiernan distintos métodos núcleo, tales como clasificadores, métodos de regresión o incluso las medidas de independencia estadística, como propagar mejor los errores y las distorsiones de los datos de entrada con procesos gaussianos, así como dónde y cuándo se puede encontrar más información en cubos arbitrarios espacio-temporales. Las técnicas presentadas abren una amplia gama de posibles casos de uso y de aplicaciones, con las que prevemos un uso más extenso y robusto de algoritmos estadísticos en las ciencias de la Tierra y el clima.Machine learning has made great strides in today's Science and engineering in general and Earth Sciences in particular. However, Earth data poses particularly challenging problems for machine learning due to not only the volume of data, but also the spatial-temporal nonlinear correlations, noise and uncertainty sources, and heterogeneous sources of information. More data does not necessarily imply more information. Therefore, extracting knowledge and information content using data analysis and modeling is important and is especially prevalent in an era where data volume and heterogeneity is steadily increasing. This calls for advances in methods that can quantify information and characterize distributions accurately. Quantifying information content within our system's data and models are still unresolved problems in statistics and machine learning. This thesis introduces new machine learning models to extract knowledge and information from Earth data. We propose kernel methods, Gaussian processes and multivariate Gaussianization to handle uncertainty and information quantification and we apply these methods to a wide range of Earth system science problems. These involve many types of learning problems including classification, regression, density estimation, synthesis, error propagation and information-theoretic measures estimation. We also demonstrate how these methods perform with different data sources including sensory data (radar, multispectral, hyperspectral, infrared sounders), data products (observations, reanalysis and model simulations) and data cubes (aggregates of various spatial-temporal data sources). The presented methodologies allow us to quantify and visualize what are the salient features driving kernel classifiers, regressors or dependence measures, how to better propagate errors and distortions of input data with Gaussian processes, and where and when more information can be found in arbitrary spatial-temporal data cubes. The presented techniques open a wide range of possible use cases and applications and we anticipate a wider adoption in the Earth sciences

A quantitative and qualitative approach to cuttlefish (Sepia officinalis) body patterning

Cuttlefish are renowned for their ability to quickly alter the colour and texture of their skin, for camouflage and communication. This is due to the presence of thousands of pigment-filled sacs, known as chromatophores, which are distributed across the skin. The chromatophores are innervated by motoneurons, which dilate the chromatophores to create the spots, stripes, and other markings, known as chromatic components. There are 34 recognized chromatic components, and it is an interesting question how cuttlefish coordinate the expression of these components to camouflage and communicate. The digital age has introduced new, powerful algorithms and methods to tease out subtle features in the coloration patterns, by means of image registration, segmentation, and identification, as well as methods for modeling the underlying control systems. These tools offer major new insights into the mechanisms of visual perception. In addition, powerful techniques have recently been developed that have yet to be applied to this complex visual motor control system. These methods have large potential in helping discover what features between the pattern and the environment are necessary to prevent detection. Here I present four laboratory experiments, that for the first time use machine learning models, to investigate cuttlefish pattern formation, implementation, and information. The first two experimental chapters investigate how cuttlefish orchestrate their chromatic components for camouflage patterns, and what strategies they employ on diverse backgrounds. I demonstrate that components are expressed more independently than previously believed, finding that the range of patterns expressed lie on a continuum, allowing us to suggest a revised classification scheme for cuttlefish body patterns. The diversity of patterns seem to imply that a cuttlefish could use its repertoire flexibly to display the maximally cryptic pattern for a given background, however I show that cuttlefish to not in fact select a single (possibly optimal) camouflage pattern, continually alter their appearance on a given background, and that the frequency of change increases in relation to the size of the objects in the environment. My third chapter investigated the language-like properties of cuttlefish communication using human speech recognition models. From our subset of cuttlefish patterns, I discovered cuttlefish utilize a lexicon of 10 patterns, with language-like properties such as: they obeyed Zipf’s law, contained around 1.6 bits per display, and interestingly, while 2 patterns were visually similar, they were displayed in separate contexts. By implementing a regression onto the patterns, I introduce a basic dictionary of cuttlefish terms and their meaning. From my investigations into cuttlefish intraspecific signaling, I discovered two previously undocumented patterns, used in agonistic encounters between cuttlefish. My final chapter describes these patterns and the contexts they are displayed

Deciphering Surfaces of Trans-Neptunian and Kuiper Belt Objects using Radiative Scattering Models, Machine Learning, and Laboratory Experiments

Decoding surface-atmospheric interactions and volatile transport mechanisms on trans-Neptunian objects (TNOs) and Kuiper Belt objects (KBOs) involves an in-depth understanding of physical and thermal properties and spatial distribution of surface constituents – nitrogen (N2), methane (CH4), carbon monoxide (CO), and water (H2O) ices. This thesis implements a combination of radiative scattering models, machine learning techniques, and laboratory experiments to investigate the uncertainties in grain size estimation of ices, the spatial distribution of surface compositions on Pluto, and the thermal properties of volatiles found on TNOs and KBOs. Radiative scattering models (Mie theory and Hapke approximations) were used to compare single scattering albedos of N2, CH4, and H2O ices from their optical constants at near-infrared wavelengths (1 – 5 µm). Based on the results of Chapters 2 and 3, this thesis recommends using the Mie model for unknown spectra of outer solar system bodies in estimating grain sizes of surface ices. When using an approximation for radiative transfer models (RTMs), we recommend using the Hapke slab approximation model over the internal scattering model. In Chapter 4, this thesis utilizes near-infrared (NIR) spectral observations of the LEISA/Ralph instrument onboard NASA’s New Horizons spacecraft. Hyperspectral LEISA data were used to map the geographic distribution of ices on Pluto’s surface by implementing the principal component reduced Gaussian mixture model (PC-GMM), an unsupervised machine learning technique. The distribution of ices reveals a latitudinal pattern with distinct surface compositions of volatiles. The PC-GMM method was able to recognize local-scale variations in surface compositions of geological features. The mapped distribution of surface units and their compositions are consistent with existing literature and help in an improved understanding of the volatile transport mechanism on the dwarf planet. In Chapter 5, we propose a method to estimate thermal conductivity, volumetric heat capacity, thermal diffusivity, and thermal inertia of N2, CH4, and CO ices, and mixtures thereof in a simulated laboratory setting at temperatures of 20 to 60 K – relevant to TNOs and KBOs. A new laboratory experimental facility – named the Outer Solar System Astrophysics Lab (OSSAL) – was built to implement the proposed method. This thesis provides detailed technical specifications of that laboratory with an emphasis on facilitating the design of similar cryogenic facilities in the future. Thus, this research was able to incorporate a set of methods, tools, and techniques for an improved understanding of ices found in the Kuiper Belt and to decipher surface-atmospheric interactions and volatile transport mechanisms on planetary bodies in the outer solar system

Deciphering Surfaces of Trans-Neptunian and Kuiper Belt Objects using Radiative Scattering Models, Machine Learning, and Laboratory Experiments

Decoding surface-atmospheric interactions and volatile transport mechanisms on trans-Neptunian objects (TNOs) and Kuiper Belt objects (KBOs) involves an in-depth understanding of physical and thermal properties and spatial distribution of surface constituents – nitrogen (N2), methane (CH4), carbon monoxide (CO), and water (H2O) ices. This thesis implements a combination of radiative scattering models, machine learning techniques, and laboratory experiments to investigate the uncertainties in grain size estimation of ices, the spatial distribution of surface compositions on Pluto, and the thermal properties of volatiles found on TNOs and KBOs. Radiative scattering models (Mie theory and Hapke approximations) were used to compare single scattering albedos of N2, CH4, and H2O ices from their optical constants at near-infrared wavelengths (1 – 5 µm). Based on the results of Chapters 2 and 3, this thesis recommends using the Mie model for unknown spectra of outer solar system bodies in estimating grain sizes of surface ices. When using an approximation for radiative transfer models (RTMs), we recommend using the Hapke slab approximation model over the internal scattering model. In Chapter 4, this thesis utilizes near-infrared (NIR) spectral observations of the LEISA/Ralph instrument onboard NASA’s New Horizons spacecraft. Hyperspectral LEISA data were used to map the geographic distribution of ices on Pluto’s surface by implementing the principal component reduced Gaussian mixture model (PC-GMM), an unsupervised machine learning technique. The distribution of ices reveals a latitudinal pattern with distinct surface compositions of volatiles. The PC-GMM method was able to recognize local-scale variations in surface compositions of geological features. The mapped distribution of surface units and their compositions are consistent with existing literature and help in an improved understanding of the volatile transport mechanism on the dwarf planet. In Chapter 5, we propose a method to estimate thermal conductivity, volumetric heat capacity, thermal diffusivity, and thermal inertia of N2, CH4, and CO ices, and mixtures thereof in a simulated laboratory setting at temperatures of 20 to 60 K – relevant to TNOs and KBOs. A new laboratory experimental facility – named the Outer Solar System Astrophysics Lab (OSSAL) – was built to implement the proposed method. This thesis provides detailed technical specifications of that laboratory with an emphasis on facilitating the design of similar cryogenic facilities in the future. Thus, this research was able to incorporate a set of methods, tools, and techniques for an improved understanding of ices found in the Kuiper Belt and to decipher surface-atmospheric interactions and volatile transport mechanisms on planetary bodies in the outer solar system

Image-set, Temporal and Spatiotemporal Representations of Videos for Recognizing, Localizing and Quantifying Actions

This dissertation addresses the problem of learning video representations, which is defined here as transforming the video so that its essential structure is made more visible or accessible for action recognition and quantification. In the literature, a video can be represented by a set of images, by modeling motion or temporal dynamics, and by a 3D graph with pixels as nodes. This dissertation contributes in proposing a set of models to localize, track, segment, recognize and assess actions such as (1) image-set models via aggregating subset features given by regularizing normalized CNNs, (2) image-set models via inter-frame principal recovery and sparsely coding residual actions, (3) temporally local models with spatially global motion estimated by robust feature matching and local motion estimated by action detection with motion model added, (4) spatiotemporal models 3D graph and 3D CNN to model time as a space dimension, (5) supervised hashing by jointly learning embedding and quantization, respectively. State-of-the-art performances are achieved for tasks such as quantifying facial pain and human diving. Primary conclusions of this dissertation are categorized as follows: (i) Image set can capture facial actions that are about collective representation; (ii) Sparse and low-rank representations can have the expression, identity and pose cues untangled and can be learned via an image-set model and also a linear model; (iii) Norm is related with recognizability; similarity metrics and loss functions matter; (v) Combining the MIL based boosting tracker with the Particle Filter motion model induces a good trade-off between the appearance similarity and motion consistence; (iv) Segmenting object locally makes it amenable to assign shape priors; it is feasible to learn knowledge such as shape priors online from Web data with weak supervision; (v) It works locally in both space and time to represent videos as 3D graphs; 3D CNNs work effectively when inputted with temporally meaningful clips; (vi) the rich labeled images or videos help to learn better hash functions after learning binary embedded codes than the random projections. In addition, models proposed for videos can be adapted to other sequential images such as volumetric medical images which are not included in this dissertation

Visible and Near Infrared imaging spectroscopy and the exploration of small scale hydrothermally altered and hydrated environments on Earth and Mars

The use of Visible and Near Infrared (VNIR) imaging spectroscopy is a cornerstone of planetary exploration. This work shall present an investigation into the limitations of scale, both spectral and spatial, in the utility of VNIR images for identifying small scale hydrothermal and potential hydrated environments on Mars, and regions of the Earth that can serve as martian analogues. Such settings represent possible habitable environments; important locations for astrobiological research. The ESA/Roscosmos ExoMars rover PanCam captures spectrally coarse but spatially high resolution VNIR images. This instrument is still in development and the first field trial of an emulator fitted with the final set of geological filters is presented here. Efficient image analysis techniques are explored and the ability to accurately characterise a hydrothermally altered region using PanCam data products is established. The CRISM orbital instrument has been returning hyperspectral VNIR images with an 18 m2 pixel resolution since 2006. The extraction of sub-pixel information from CRISM pixels using Spectral Mixture Analysis (SMA) algorithms is explored. Using synthetic datasets a full SMA pipeline consisting of publically available Matlab algorithms and optimised for investigation of mineralogically complex hydrothermal suites is developed for the first time. This is validated using data from Námafjall in Iceland, the region used to field trial the PanCam prototype. The pipeline is applied to CRISM images covering four regions on Mars identified as having potentially undergone hydrothermal alteration in their past. A second novel use of SMA to extract a unique spectral signature for the potentially hydrated Recurring Slope Lineae features on Mars is presented. The specific methodology presented shows promise and future improvements are suggested. The importance of combining different scales of data and recognising their limitations is discussed based on the results presented and ways in which to take the results presented in this thesis forward are given