Hyperspectral Unmixing Overview: Geometrical, Statistical, and Sparse Regression-Based Approaches

Imaging spectrometers measure electromagnetic energy scattered in their instantaneous field view in hundreds or thousands of spectral channels with higher spectral resolution than multispectral cameras. Imaging spectrometers are therefore often referred to as hyperspectral cameras (HSCs). Higher spectral resolution enables material identification via spectroscopic analysis, which facilitates countless applications that require identifying materials in scenarios unsuitable for classical spectroscopic analysis. Due to low spatial resolution of HSCs, microscopic material mixing, and multiple scattering, spectra measured by HSCs are mixtures of spectra of materials in a scene. Thus, accurate estimation requires unmixing. Pixels are assumed to be mixtures of a few materials, called endmembers. Unmixing involves estimating all or some of: the number of endmembers, their spectral signatures, and their abundances at each pixel. Unmixing is a challenging, ill-posed inverse problem because of model inaccuracies, observation noise, environmental conditions, endmember variability, and data set size. Researchers have devised and investigated many models searching for robust, stable, tractable, and accurate unmixing algorithms. This paper presents an overview of unmixing methods from the time of Keshava and Mustard's unmixing tutorial [1] to the present. Mixing models are first discussed. Signal-subspace, geometrical, statistical, sparsity-based, and spatial-contextual unmixing algorithms are described. Mathematical problems and potential solutions are described. Algorithm characteristics are illustrated experimentally.Comment: This work has been accepted for publication in IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensin

A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote Sensin

Cross-Attention in Coupled Unmixing Nets for Unsupervised Hyperspectral Super-Resolution

The recent advancement of deep learning techniques has made great progress on hyperspectral image super-resolution (HSI-SR). Yet the development of unsupervised deep networks remains challenging for this task. To this end, we propose a novel coupled unmixing network with a cross-attention mechanism, CUCaNet for short, to enhance the spatial resolution of HSI by means of higher-spatial-resolution multispectral image (MSI). Inspired by coupled spectral unmixing, a two-stream convolutional autoencoder framework is taken as backbone to jointly decompose MS and HS data into a spectrally meaningful basis and corresponding coefficients. CUCaNet is capable of adaptively learning spectral and spatial response functions from HS-MS correspondences by enforcing reasonable consistency assumptions on the networks. Moreover, a cross-attention module is devised to yield more effective spatial-spectral information transfer in networks. Extensive experiments are conducted on three widely-used HS-MS datasets in comparison with state-of-the-art HSI-SR models, demonstrating the superiority of the CUCaNet in the HSI-SR application. Furthermore, the codes and datasets will be available at: https://github.com/danfenghong/ECCV2020_CUCaNet

Linear unmixing of multidate hyperspectral imagery for crop yield estimation

International audienc

Dynamical Hyperspectral Unmixing with Variational Recurrent Neural Networks

Multitemporal hyperspectral unmixing (MTHU) is a fundamental tool in the analysis of hyperspectral image sequences. It reveals the dynamical evolution of the materials (endmembers) and of their proportions (abundances) in a given scene. However, adequately accounting for the spatial and temporal variability of the endmembers in MTHU is challenging, and has not been fully addressed so far in unsupervised frameworks. In this work, we propose an unsupervised MTHU algorithm based on variational recurrent neural networks. First, a stochastic model is proposed to represent both the dynamical evolution of the endmembers and their abundances, as well as the mixing process. Moreover, a new model based on a low-dimensional parametrization is used to represent spatial and temporal endmember variability, significantly reducing the amount of variables to be estimated. We propose to formulate MTHU as a Bayesian inference problem. However, the solution to this problem does not have an analytical solution due to the nonlinearity and non-Gaussianity of the model. Thus, we propose a solution based on deep variational inference, in which the posterior distribution of the estimated abundances and endmembers is represented by using a combination of recurrent neural networks and a physically motivated model. The parameters of the model are learned using stochastic backpropagation. Experimental results show that the proposed method outperforms state of the art MTHU algorithms

Combining hyperspectral UAV and mulitspectral FORMOSAT-2 imagery for precision agriculture applications

Precision agriculture requires detailed information regarding the crop status variability within a field. Remote sensing provides an efficient way to obtain such information through observing biophysical parameters, such as canopy nitrogen content, leaf coverage, and plant biomass. However, individual remote sensing sensors often fail to provide information which meets the spatial and temporal resolution required by precision agriculture. The purpose of this study is to investigate methods which can be used to combine imagery from various sensors in order to create a new dataset which comes closer to meeting these requirements. More specifically, this study combined multispectral satellite imagery (Formosat-2) and hyperspectral Unmanned Aerial Vehicle (UAV) imagery of a potato field in the Netherlands. The imagery from both platforms was combined in two ways. Firstly, data fusion methods brought the spatial resolution of the Formosat-2 imagery (8 m) down to the spatial resolution of the UAV imagery (1 m). Two data fusion methods were applied: an unmixing-based algorithm and the Spatial and Temporal Adaptive Reflectance Fusion Model (STARFM). The unmixing-based method produced vegetation indices which were highly correlated to the measured LAI (rs= 0.866) and canopy chlorophyll values (rs=0.884), whereas the STARFM obtained lower correlations. Secondly, a Spectral-Temporal Reflectance Surface (STRS) was constructed to interpolate a daily 101 band reflectance spectra using both sources of imagery. A novel STRS method was presented, which utilizes Bayesian theory to obtain realistic spectra and accounts for sensor uncertainties. The resulting surface obtained a high correlation to LAI (rs=0.858) and canopy chlorophyll (rs=0.788) measurements at field level. The multi-sensor datasets were able to characterize significant differences of crop status due to differing nitrogen fertilization regimes from June to August. Meanwhile, the yield prediction models based purely on the vegetation indices extracted from the unmixing-based fusion dataset explained 52.7% of the yield variation, whereas the STRS dataset was able to explain 72.9% of the yield variability. The results of the current study indicate that the limitations of each individual sensor can be largely surpassed by combining multiple sources of imagery, which is beneficial for agricultural management. Further research could focus on the integration of data fusion and STRS techniques, and the inclusion of imagery from additional sensors.Samenvatting In een wereld waar toekomstige voedselzekerheid bedreigd wordt, biedt precisielandbouw een oplossing die de oogst kan maximaliseren terwijl de economische en ecologische kosten van voedselproductie beperkt worden. Om dit te kunnen doen is gedetailleerde informatie over de staat van het gewas nodig. Remote sensing is een manier om biofysische informatie, waaronder stikstof gehaltes en biomassa, te verkrijgen. De informatie van een individuele sensor is echter vaak niet genoeg om aan de hoge eisen betreft ruimtelijke en temporele resolutie te voldoen. Deze studie combineert daarom de informatie afkomstig van verschillende sensoren, namelijk multispectrale satelliet beelden (Formosat-2) en hyperspectral Unmanned Aerial Vehicle (UAV) beelden van een aardappel veld, in een poging om aan de hoge informatie eisen van precisielandbouw te voldoen. Ten eerste werd gebruik gemaakt van datafusie om de acht Formosat-2 beelden met een resolutie van 8 m te combineren met de vier UAV beelden met een resolutie van 1 m. De resulterende dataset bestaat uit acht beelden met een resolutie van 1 m. Twee methodes werden toegepast, de zogenaamde STARFM methode en een unmixing-based methode. De unmixing-based methode produceerde beelden met een hoge correlatie op de Leaf Area Index (LAI) (rs= 0.866) en chlorofyl gehalte (rs=0.884) gemeten op veldnieveau. De STARFM methode presteerde slechter, met correlaties van respectievelijk rs=0.477 en rs=0.431. Ten tweede werden Spectral-Temporal Reflectance Surfaces (STRSs) ontwikkeld die een dagelijks spectrum weergeven met 101 spectrale banden. Om dit te doen is een nieuwe STRS methode gebaseerd op de Bayesiaanse theorie ontwikkeld. Deze produceert realistische spectra met een overeenkomstige onzekerheid. Deze STRSs vertoonden hoge correlaties met de LAI (rs=0.858) en het chlorofyl gehalte (rs=0.788) gemeten op veldnieveau. De bruikbaarheid van deze twee soorten datasets werd geanalyseerd door middel van de berekening van een aantal vegetatie-indexen. De resultaten tonen dat de multi-sensor datasets capabel zijn om significante verschillen in de groei van gewassen vast te stellen tijdens het groeiseizoen zelf. Bovendien werden regressiemodellen toegepast om de bruikbaarheid van de datasets voor oogst voorspellingen. De unmixing-based datafusie verklaarde 52.7% van de variatie in oogst, terwijl de STRS 72.9% van de variabiliteit verklaarden. De resultaten van het huidige onderzoek tonen aan dat de beperkingen van een individuele sensor grotendeels overtroffen kunnen worden door het gebruik van meerdere sensoren. Het combineren van verschillende sensoren, of het nu Formosat-2 en UAV beelden zijn of andere ruimtelijke informatiebronnen, kan de hoge informatie eisen van de precisielandbouw tegemoet komen.In the context of threatened global food security, precision agriculture is one strategy to maximize yield to meet the increased demands of food, while minimizing both economic and environmental costs of food production. This is done by applying variable management strategies, which means the fertilizer or irrigation rates within a field are adjusted according to the crop needs in that specific part of the field. This implies that accurate crop status information must be available regularly for many different points in the field. Remote sensing can provide this information, but it is difficult to meet the information requirements when using only one sensor. For example, satellites collect imagery regularly and over large areas, but may be blocked by clouds. Unmanned Aerial Vehicles (UAVs), commonly known as drones, are more flexible but have higher operational costs. The purpose of this study was to use fusion methods to combine satellite (Formosat-2) with UAV imagery of a potato field in the Netherlands. Firstly, data fusion was applied. The eight Formosat-2 images with 8 m x 8 m pixels were combined with four UAV images with 1 m x 1 m pixels to obtain a new dataset of eight images with 1 m x 1 m pixels. Unmixing-based data fusion produced images which had a high correlation to field measurements obtained from the potato field during the growing season. The results of a second data fusion method, STARFM, were less reliable in this study. The UAV images were hyperspectral, meaning they contained very detailed information spanning a large part of the electromagnetic spectrum. Much of this information was lost in the data fusion methods because the Formosat-2 images were multispectral, representing a more limited portion of the spectrum. Therefore, a second analysis investigated the use of Spectral-Temporal Reflectance Surfaces (STRS), which allow information from different portions of the electromagnetic spectrum to be combined. These STRS provided daily hyperspectral observations, which were also verified as accurate by comparing them to reference data. Finally, this study demonstrated the ability of both data fusion and STRS to identify which parts of the potato field had lower photosynthetic production during the growing season. Data fusion was capable of explaining 52.7% of the yield variation through regression models, whereas the STRS explained 72.9%. To conclude, this study indicates how to combine crop status information from different sensors to support precision agriculture management decisions

Unsupervised Hyperspectral and Multispectral Images Fusion Based on the Cycle Consistency

Hyperspectral images (HSI) with abundant spectral information reflected materials property usually perform low spatial resolution due to the hardware limits. Meanwhile, multispectral images (MSI), e.g., RGB images, have a high spatial resolution but deficient spectral signatures. Hyperspectral and multispectral image fusion can be cost-effective and efficient for acquiring both high spatial resolution and high spectral resolution images. Many of the conventional HSI and MSI fusion algorithms rely on known spatial degradation parameters, i.e., point spread function, spectral degradation parameters, spectral response function, or both of them. Another class of deep learning-based models relies on the ground truth of high spatial resolution HSI and needs large amounts of paired training images when working in a supervised manner. Both of these models are limited in practical fusion scenarios. In this paper, we propose an unsupervised HSI and MSI fusion model based on the cycle consistency, called CycFusion. The CycFusion learns the domain transformation between low spatial resolution HSI (LrHSI) and high spatial resolution MSI (HrMSI), and the desired high spatial resolution HSI (HrHSI) are considered to be intermediate feature maps in the transformation networks. The CycFusion can be trained with the objective functions of marginal matching in single transform and cycle consistency in double transforms. Moreover, the estimated PSF and SRF are embedded in the model as the pre-training weights, which further enhances the practicality of our proposed model. Experiments conducted on several datasets show that our proposed model outperforms all compared unsupervised fusion methods. The codes of this paper will be available at this address: https: //github.com/shuaikaishi/CycFusion for reproducibility

Assessing the role of EO in biodiversity monitoring: options for integrating in-situ observations with EO within the context of the EBONE concept

The European Biodiversity Observation Network (EBONE) is a European contribution on terrestrial monitoring to GEO BON, the Group on Earth Observations Biodiversity Observation Network. EBONE’s aims are to develop a system of biodiversity observation at regional, national and European levels by assessing existing approaches in terms of their validity and applicability starting in Europe, then expanding to regions in Africa. The objective of EBONE is to deliver: 1. A sound scientific basis for the production of statistical estimates of stock and change of key indicators; 2. The development of a system for estimating past changes and forecasting and testing policy options and management strategies for threatened ecosystems and species; 3. A proposal for a cost-effective biodiversity monitoring system. There is a consensus that Earth Observation (EO) has a role to play in monitoring biodiversity. With its capacity to observe detailed spatial patterns and variability across large areas at regular intervals, our instinct suggests that EO could deliver the type of spatial and temporal coverage that is beyond reach with in-situ efforts. Furthermore, when considering the emerging networks of in-situ observations, the prospect of enhancing the quality of the information whilst reducing cost through integration is compelling. This report gives a realistic assessment of the role of EO in biodiversity monitoring and the options for integrating in-situ observations with EO within the context of the EBONE concept (cfr. EBONE-ID1.4). The assessment is mainly based on a set of targeted pilot studies. Building on this assessment, the report then presents a series of recommendations on the best options for using EO in an effective, consistent and sustainable biodiversity monitoring scheme. The issues that we faced were many: 1. Integration can be interpreted in different ways. One possible interpretation is: the combined use of independent data sets to deliver a different but improved data set; another is: the use of one data set to complement another dataset. 2. The targeted improvement will vary with stakeholder group: some will seek for more efficiency, others for more reliable estimates (accuracy and/or precision); others for more detail in space and/or time or more of everything. 3. Integration requires a link between the datasets (EO and in-situ). The strength of the link between reflected electromagnetic radiation and the habitats and their biodiversity observed in-situ is function of many variables, for example: the spatial scale of the observations; timing of the observations; the adopted nomenclature for classification; the complexity of the landscape in terms of composition, spatial structure and the physical environment; the habitat and land cover types under consideration. 4. The type of the EO data available varies (function of e.g. budget, size and location of region, cloudiness, national and/or international investment in airborne campaigns or space technology) which determines its capability to deliver the required output. EO and in-situ could be combined in different ways, depending on the type of integration we wanted to achieve and the targeted improvement. We aimed for an improvement in accuracy (i.e. the reduction in error of our indicator estimate calculated for an environmental zone). Furthermore, EO would also provide the spatial patterns for correlated in-situ data. EBONE in its initial development, focused on three main indicators covering: (i) the extent and change of habitats of European interest in the context of a general habitat assessment; (ii) abundance and distribution of selected species (birds, butterflies and plants); and (iii) fragmentation of natural and semi-natural areas. For habitat extent, we decided that it did not matter how in-situ was integrated with EO as long as we could demonstrate that acceptable accuracies could be achieved and the precision could consistently be improved. The nomenclature used to map habitats in-situ was the General Habitat Classification. We considered the following options where the EO and in-situ play different roles: using in-situ samples to re-calibrate a habitat map independently derived from EO; improving the accuracy of in-situ sampled habitat statistics, by post-stratification with correlated EO data; and using in-situ samples to train the classification of EO data into habitat types where the EO data delivers full coverage or a larger number of samples. For some of the above cases we also considered the impact that the sampling strategy employed to deliver the samples would have on the accuracy and precision achieved. Restricted access to European wide species data prevented work on the indicator ‘abundance and distribution of species’. With respect to the indicator ‘fragmentation’, we investigated ways of delivering EO derived measures of habitat patterns that are meaningful to sampled in-situ observations