A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote Sensin

Underwater target detection with hyperspectral data : solutions for both known and unknown water quality", S. Jay, M. Guillaume, J. Blanc-Talon, , IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 5 :1213-1221, 2012. IF 2.87

International audienceIn this paper, we present various bathymetric filters, based on the well-known MF, AMF and ACE detectors, for underwater target detection from hyperspectral remote-sensing data. In the case of unknown water characteristics, we also propose the GBF, a GLRT-based filter that estimates these parameters and detects at the same time. The results of this estimation process, performed on both simulated and real data, are encouraging, since under regular conditions of depth, water quality and SNR, the accuracy is quite good. We show that these new detectors outperform the usual ones, obtained by detecting after correction of the water column effect by a classical method. We also show that the estimation errors do not impact much the detection performances, and therefore, this underwater target detection method is self-sufficient and can be implemented without any a priori knowledge on the water column

Light Field and Water Clarity Simulation of Natural Environments in Laboratory Conditions

Simulation of natural oceanic conditions in a laboratory setting is a challenging task, especially when that environment can be miles away. We present an attempt to replicate the solar radiation expected at different latitudes with varying water clarity conditions up to 30 m in depth using a 2.5 m deep engineering tank at the University of New Hampshire. The goals of the study were: 1) to configure an underwater light source that produced an irradiance spectrum similar to natural daylight with the sun at zenith and at 60° under clear atmospheric conditions, and 2) to monitor water clarity as a function of depth. Irradiance was measured using a spectra-radiometer with a cosine receiver to analyze the output spectrum of submersed lamps as a function of distance. In addition, an underwater reflection method was developed to measure the diffuse attenuation coefficient in real time. Two water clarity types were characterized, clear waters representing deep, open-ocean conditions, and murky waters representing littoral environments. Results showed good correlation between the irradiance measured at 400 nm to 600 nm and the natural daylight spectrum at 3 m from the light source. This can be considered the water surface conditions reference. Using these methodologies in a controlled laboratory setting, we are able to replicate illumination and water conditions to study the physical, chemical and biological processes on natural and man-made objects and/or systems in simulated, varied geographic locations and environments

Macroalgae and eelgrass mapping in Great Bay Estuary using AISA hyperspectral imagery

Increase in nitrogen concentration and declining eelgrass beds in Great Bay Estuary have been observed in the last decades. These two parameters are clear indicators of the impending problems for NH’s estuaries. The NH Department of Environmental Services (DES) in collaboration with the New Hampshire Estuaries Project (NHEP) adopted the assumption that eelgrass survival can be used as the water quality target for nutrient criteria development for NH’s estuaries. One of the hypotheses put forward regarding eelgrass decline is that a possible eutrophication response to nutrient increases in the Great Bay Estuary has been the proliferation of nuisance macroalgae, which has reduced eelgrass area in Great Bay Estuary. To test this hypothesis, mapping of eelgrass and nuisance macroalgae beds using hyperspectral imagery was suggested. A hyperspectral imagery was conducted by SpecTIR in August 2007 using an AISA Eagle sensor. The collected dataset was used to map eelgrass and nuisance macroalgae throughout the Great Bay Estuary. This report outlines the configured procedure for mapping the macroalgae and eelgrass beds using hyperspectral imagery. No ground truth measurements of eelgrass or macroalgae were collected as part of this project, although eelgrass ground truth data was collected as part of a separate project. Guidance from eelgrass and macroalgae experts was used for identifying training sets and evaluating the classification results. The results produced a comprehensive eelgrass and macroalgae map of the estuary. Three recommendations are suggested following the experience gained in this study: conducting ground truth measurements at the time of the HS survey, acquiring the current DEM model of Great Bay Estuary, and examining additional HS datasets with expert eelgrass and macroalgae guidance. These three issues can improve the classification results and allow more advanced applications, such as identification of macroalgae types

Hyperspectral benthic mapping from underwater robotic platforms

We live on a planet of vast oceans; 70% of the Earth's surface is covered in water. They are integral to supporting life, providing 99% of the inhabitable space on Earth. Our oceans and the habitats within them are under threat due to a variety of factors. To understand the impacts and possible solutions, the monitoring of marine habitats is critically important. Optical imaging as a method for monitoring can provide a vast array of information however imaging through water is complex. To compensate for the selective attenuation of light in water, this thesis presents a novel light propagation model and illustrates how it can improve optical imaging performance. An in-situ hyperspectral system is designed which comprised of two upward looking spectrometers at different positions in the water column. The downwelling light in the water column is continuously sampled by the system which allows for the generation of a dynamic water model. In addition to the two upward looking spectrometers the in-situ system contains an imaging module which can be used for imaging of the seafloor. It consists of a hyperspectral sensor and a trichromatic stereo camera. New calibration methods are presented for the spatial and spectral co-registration of the two optical sensors. The water model is used to create image data which is invariant to the changing optical properties of the water and changing environmental conditions. In this thesis the in-situ optical system is mounted onboard an Autonomous Underwater Vehicle. Data from the imaging module is also used to classify seafloor materials. The classified seafloor patches are integrated into a high resolution 3D benthic map of the surveyed site. Given the limited imaging resolution of the hyperspectral sensor used in this work, a new method is also presented that uses information from the co-registered colour images to inform a new spectral unmixing method to resolve subpixel materials

Restoration of Oyster (Crassostrea virginica) Habitat for Multiple Estuarine Species Benefits

Increase in nitrogen concentration and declining eelgrass beds in Great Bay Estuary have been observed in the last decades. These two parameters are clear indicators of the impending problems for NH’s estuaries. The NH Department of Environmental Services (DES) in collaboration with the New Hampshire Estuaries Project (NHEP) adopted the assumption that eelgrass survival can be used as the water quality target for nutrient criteria development for NH’s estuaries. One of the hypotheses put forward regarding eelgrass decline is that a possible eutrophication response to nutrient increases in the Great Bay Estuary has been the proliferation of nuisance macroalgae, which has reduced eelgrass area in Great Bay Estuary. To test this hypothesis, mapping of eelgrass and nuisance macroalgae beds using hyperspectral imagery was suggested. A hyperspectral imagery was conducted by SpecTIR in August 2007 using an AISA Eagle sensor. The collected dataset was used to map eelgrass and nuisance macroalgae throughout the Great Bay Estuary. This report outlines the configured procedure for mapping the macroalgae and eelgrass beds using hyperspectral imagery. No ground truth measurements of eelgrass or macroalgae were collected as part of this project, although eelgrass ground truth data was collected as part of a separate project. Guidance from eelgrass and macroalgae experts was used for identifying training sets and evaluating the classification results. The results produced a comprehensive eelgrass and macroalgae map of the estuary. Three recommendations are suggested following the experience gained in this study: conducting ground truth measurements at the time of the HS survey, acquiring the current DEM model of Great Bay Estuary, and examining additional HS datasets with expert eelgrass and macroalgae guidance. These three issues can improve the classification results and allow more advanced applications, such as identification of macroalgae types

Using Moored Arrays and Hyperspectral Aerial Imagery to Develop Nutrient Criteria for New Hampshire\u27s Estuaries

Increasing nitrogen concentrations and declining eelgrass beds in Great Bay, NH are clear indicators of impending problems for the state’s estuaries. A workgroup established in 2005 by the NH Department of Environmental Services and the NH Estuaries Project (NHEP) adopted eelgrass survival as the water quality target for nutrient criteria development for NH’s estuaries. In 2007, the NHEP received a grant from the U.S. Environmental Protection Agency to collect water quality information including that from moored sensors and hyper-spectral imagery data of the Great Bay Estuary. Data from the Great Bay Coastal Buoy, part of the regional Integrated Ocean Observing System (IOOS), were used to derive a multivariate model of water clarity with phytoplankton, Colored Dissolved Organic Matter (CDOM), and non-algal particles. Non-algal particles include both inorganic and organic matter. Most of the temporal variability in the diffuse attenuation coefficient of Photosynthetically Available Radiation (PAR) was associated with non-algal particles. However, on a mean daily basis non-algal particles and CDOM contributed a similar fraction (~30 %) to the attenuation of light. The contribution of phytoplankton was about a third of the other two optically important constituents. CDOM concentrations varied with salinity and magnitude of riverine inputs demonstrating its terrestrial origin. Non-algal particle concentration also varied with river flow but also wind driven resuspension. Twelve of the NHEP estuarine assessment zones were observed with the hyperspectral aerial imagery on August 29 and October 17. A concurrent in situ effort included buoy measurements, continuous along-track sampling, discrete water grab samples, and vertical profiles of light attenuation. PAR effective attenuation coefficients retrieved from deep water regions in the imagery agreed well with in-situ observations. Water clarity was lower and optically important constituent concentrations were higher in the tributaries. Eelgrass survival depth, estimated as the depth at which 22% of surface light was available, ranged from less than half a meter to over two meters. The best water clarity was found in the Great Bay (GB), Little Bay (LB), and Lower Piscataqua River (LPR) assessment zones. Absence of eelgrass from these zones would indicate controlling factors other than water clarity

Hyperspectral Imaging for Fine to Medium Scale Applications in Environmental Sciences

The aim of the Special Issue “Hyperspectral Imaging for Fine to Medium Scale Applications in Environmental Sciences” was to present a selection of innovative studies using hyperspectral imaging (HSI) in different thematic fields. This intention reflects the technical developments in the last three decades, which have brought the capacity of HSI to provide spectrally, spatially and temporally detailed data, favoured by e.g., hyperspectral snapshot technologies, miniaturized hyperspectral sensors and hyperspectral microscopy imaging. The present book comprises a suite of papers in various fields of environmental sciences—geology/mineral exploration, digital soil mapping, mapping and characterization of vegetation, and sensing of water bodies (including under-ice and underwater applications). In addition, there are two rather methodically/technically-oriented contributions dealing with the optimized processing of UAV data and on the design and test of a multi-channel optical receiver for ground-based applications. All in all, this compilation documents that HSI is a multi-faceted research topic and will remain so in the future

A Trillion Coral Reef Colors: Deeply Annotated Underwater Hyperspectral Images for Automated Classification and Habitat Mapping

This paper describes a large dataset of underwater hyperspectral imagery that can be used by researchers in the domains of computer vision, machine learning, remote sensing, and coral reef ecology. We present the details of underwater data acquisition, processing and curation to create this large dataset of coral reef imagery annotated for habitat mapping. A diver-operated hyperspectral imaging system (HyperDiver) was used to survey 147 transects at 8 coral reef sites around the Caribbean island of Curacao. The underwater proximal sensing approach produced fine-scale images of the seafloor, with more than 2.2 billion points of detailed optical spectra. Of these, more than 10 million data points have been annotated for habitat descriptors or taxonomic identity with a total of 47 class labels up to genus- and species-levels. In addition to HyperDiver survey data, we also include images and annotations from traditional (color photo) quadrat surveys conducted along 23 of the 147 transects, which enables comparative reef description between two types of reef survey methods. This dataset promises benefits for efforts in classification algorithms, hyperspectral image segmentation and automated habitat mapping. Dataset: https://doi.org/10.1594/PANGAEA.911300 Dataset License: CC-BY-N