Exploiting Deep Features for Remote Sensing Image Retrieval: A Systematic Investigation

Remote sensing (RS) image retrieval is of great significant for geological information mining. Over the past two decades, a large amount of research on this task has been carried out, which mainly focuses on the following three core issues: feature extraction, similarity metric and relevance feedback. Due to the complexity and multiformity of ground objects in high-resolution remote sensing (HRRS) images, there is still room for improvement in the current retrieval approaches. In this paper, we analyze the three core issues of RS image retrieval and provide a comprehensive review on existing methods. Furthermore, for the goal to advance the state-of-the-art in HRRS image retrieval, we focus on the feature extraction issue and delve how to use powerful deep representations to address this task. We conduct systematic investigation on evaluating correlative factors that may affect the performance of deep features. By optimizing each factor, we acquire remarkable retrieval results on publicly available HRRS datasets. Finally, we explain the experimental phenomenon in detail and draw conclusions according to our analysis. Our work can serve as a guiding role for the research of content-based RS image retrieval

Deep learning in remote sensing: a review

Standing at the paradigm shift towards data-intensive science, machine learning techniques are becoming increasingly important. In particular, as a major breakthrough in the field, deep learning has proven as an extremely powerful tool in many fields. Shall we embrace deep learning as the key to all? Or, should we resist a 'black-box' solution? There are controversial opinions in the remote sensing community. In this article, we analyze the challenges of using deep learning for remote sensing data analysis, review the recent advances, and provide resources to make deep learning in remote sensing ridiculously simple to start with. More importantly, we advocate remote sensing scientists to bring their expertise into deep learning, and use it as an implicit general model to tackle unprecedented large-scale influential challenges, such as climate change and urbanization.Comment: Accepted for publication IEEE Geoscience and Remote Sensing Magazin

Feature Extraction and Design in Deep Learning Models

The selection and computation of meaningful features is critical for developing good deep learning methods. This dissertation demonstrates how focusing on this process can significantly improve the results of learning-based approaches. Specifically, this dissertation presents a series of different studies in which feature extraction and design was a significant factor for obtaining effective results. The first two studies are a content-based image retrieval system (CBIR) and a seagrass quantification study in which deep learning models were used to extract meaningful high-level features that significantly increased the performance of the approaches. Secondly, a method for change detection is proposed where the multispectral channels of satellite images are combined with different feature indices to improve the results. Then, two novel feature operators for mesh convolutional networks are presented that successfully extract invariant features from the faces and vertices of a mesh, respectively. The novel feature operators significantly outperform the previous state of the art for mesh classification and segmentation and provide two novel architectures for applying convolutional operations to the faces and vertices of geometric 3D meshes. Finally, a novel approach for automatic generation of 3D meshes is presented. The generative model efficiently uses the vertex-based feature operators proposed in the previous study and successfully learns to produce shapes from a mesh dataset with arbitrary topology

Color image registration under illumination changes

The estimation of parametric global motion has had a significant attention during the last two decades, but despite the great efforts invested, there are still open issues. One of the most important ones is related to the ability to recover large deformation between images in the presence of illumination changes while kipping accurate estimates. Illumination changes in color images are another important open issue. In this paper, a Generalized least squared-based motion estimator is used in combination with color image model to allow accurate estimates of global motion between two color images under the presence of large geometric transformation and illumination changes. Experiments using challenging images have been performed showing that the presented technique is feasible and provides accurate estimates of the motion and illumination parameter

An Integrated physics-based approach to demonstrate the potential of the Landsat Data Continuity Mission (LDCM) for monitoring coastal/inland waters

Monitoring coastal or inland waters, recognized as case II waters, using the existing Landsat technology is somewhat restricted because of its low Signal-to-Noise ratio (SNR) as well as its relatively poor radiometric resolution. As a primary task, we introduce a novel technique, which integrates the Landsat-7 data as a surrogate for LDCM with a 3D hydrodynamic model to monitor the dynamics of coastal waters near river discharges as well as in a small lake environment. The proposed approach leverages both the thermal and the reflective Landsat-7 imagery to calibrate the model and to retrieve the concentrations of optically active components of the water. To do so, the model is first calibrated by optimizing its thermal outputs with the surface temperature maps derived from the Landsat-7 data. The constituent retrieval is conducted in the second phase where multiple simulated concentration maps are provided to an in-water radiative transfer code (Hydrolight) to generate modeled surface reflectance maps. Prior to any remote sensing task, one has to ensure that a dataset comes from a well-calibrated imaging system. Although the calibration status of Landsat-7 has been regularly monitored over multiple desert sites, it was desired to evaluate its performance over dark waters relative to a well-calibrated instrument designed specifically for water studies. In the light of this, several Landsat- 7 images were cross-calibrated against the Terra-MODIS data over deep, dark waters whose optical properties remain relatively stable. This study is intended to lay the groundwork and provide a reference point for similar studies planned for the new Landsat. In an independent case study, the potential of the new Landsat sensor was examined using an EO-1 dataset and applying a spectral optimization approach over case II waters. The water constituent maps generated from the EO-1 imagery were compared against those derived from Landsat-7 to fully analyze the improvement levels pertaining to the new Landsat\u27s enhanced features in a water constituent retrieval framework

Modeling Atmosphere-Ocean Radiative Transfer: A PACE Mission Perspective

The research frontiers of radiative transfer (RT) in coupled atmosphere-ocean systems are explored to enable new science and specifically to support the upcoming Plankton, Aerosol, Cloud ocean Ecosystem (PACE) satellite mission. Given (i) the multitude of atmospheric and oceanic constituents at any given moment that each exhibits a large variety of physical and chemical properties and (ii) the diversity of light-matter interactions (scattering, absorption, and emission), tackling all outstanding RT aspects related to interpreting and/or simulating light reflected by atmosphere-ocean systems becomes impossible. Instead, we focus on both theoretical and experimental studies of RT topics important to the science threshold and goal questions of the PACE mission and the measurement capabilities of its instruments. We differentiate between (a) forward (FWD) RT studies that focus mainly on sensitivity to influencing variables and/or simulating data sets, and (b) inverse (INV) RT studies that also involve the retrieval of atmosphere and ocean parameters. Our topics cover (1) the ocean (i.e., water body): absorption and elastic/inelastic scattering by pure water (FWD RT) and models for scattering and absorption by particulates (FWD RT and INV RT); (2) the air-water interface: variations in ocean surface refractive index (INV RT) and in whitecap reflectance (INV RT); (3) the atmosphere: polarimetric and/or hyperspectral remote sensing of aerosols (INV RT) and of gases (FWD RT); and (4) atmosphere-ocean systems: benchmark comparisons, impact of the Earth's sphericity and adjacency effects on space-borne observations, and scattering in the ultraviolet regime (FWD RT). We provide for each topic a summary of past relevant (heritage) work, followed by a discussion (for unresolved questions) and RT updates

Multispectral Imaging For Face Recognition Over Varying Illumination

This dissertation addresses the advantage of using multispectral narrow-band images over conventional broad-band images for improved face recognition under varying illumination. To verify the effectiveness of multispectral images for improving face recognition performance, three sequential procedures are taken into action: multispectral face image acquisition, image fusion for multispectral and spectral band selection to remove information redundancy. Several efficient image fusion algorithms are proposed and conducted on spectral narrow-band face images in comparison to conventional images. Physics-based weighted fusion and illumination adjustment fusion make good use of spectral information in multispectral imaging process. The results demonstrate that fused narrow-band images outperform the conventional broad-band images under varying illuminations. In the case where multispectral images are acquired over severe changes in daylight, the fused images outperform conventional broad-band images by up to 78%. The success of fusing multispectral images lies in the fact that multispectral images can separate the illumination information from the reflectance of objects which is impossible for conventional broad-band images. To reduce the information redundancy among multispectral images and simplify the imaging system, distance-based band selection is proposed where a quantitative evaluation metric is defined to evaluate and differentiate the performance of multispectral narrow-band images. This method is proved to be exceptionally robust to parameter changes. Furthermore, complexity-guided distance-based band selection is proposed using model selection criterion for an automatic selection. The performance of selected bands outperforms the conventional images by up to 15%. From the significant performance improvement via distance-based band selection and complexity-guided distance-based band selection, we prove that specific facial information carried in certain narrow-band spectral images can enhance face recognition performance compared to broad-band images. In addition, both algorithms are proved to be independent to recognition engines. Significant performance improvement is achieved by proposed image fusion and band selection algorithms under varying illumination including outdoor daylight conditions. Our proposed imaging system and image processing algorithms lead to a new avenue of automatic face recognition system towards a better recognition performance than the conventional peer system over varying illuminations

Modeling atmosphere-ocean radiative transfer: A PACE mission perspective

The research frontiers of radiative transfer (RT) in coupled atmosphere-ocean systems are explored to enable new science and specifically to support the upcoming Plankton, Aerosol, Cloud ocean Ecosystem (PACE) satellite mission. Given (i) the multitude of atmospheric and oceanic constituents at any given moment that each exhibits a large variety of physical and chemical properties and (ii) the diversity of light-matter interactions (scattering, absorption, and emission), tackling all outstanding RT aspects related to interpreting and/or simulating light reflected by atmosphere-ocean systems becomes impossible. Instead, we focus on both theoretical and experimental studies of RT topics important to the science threshold and goal questions of the PACE mission and the measurement capabilities of its instruments. We differentiate between (a) forward (FWD) RT studies that focus mainly on sensitivity to influencing variables and/or simulating data sets, and (b) inverse (INV) RT studies that also involve the retrieval of atmosphere and ocean parameters. Our topics cover (1) the ocean (i.e., water body): absorption and elastic/inelastic scattering by pure water (FWD RT) and models for scattering and absorption by particulates (FWD RT and INV RT); (2) the air-water interface: variations in ocean surface refractive index (INV RT) and in whitecap reflectance (INV RT); (3) the atmosphere: polarimetric and/or hyperspectral remote sensing of aerosols (INV RT) and of gases (FWD RT); and (4) atmosphere-ocean systems: benchmark comparisons, impact of the Earth’s sphericity and adjacency effects on space-borne observations, and scattering in the ultraviolet regime (FWD RT). We provide for each topic a summary of past relevant (heritage) work, followed by a discussion (for unresolved questions) and RT updates

Modeling Atmosphere-Ocean Radiative Transfer: A PACE Mission Perspective

The research frontiers of radiative transfer (RT) in coupled atmosphere-ocean systems are explored to enable new science and specifically to support the upcoming Plankton, Aerosol, Cloud ocean Ecosystem (PACE) satellite mission. Given (i) the multitude of atmospheric and oceanic constituents at any given moment that each exhibits a large variety of physical and chemical properties and (ii) the diversity of light-matter interactions (scattering, absorption, and emission), tackling all outstanding RT aspects related to interpreting and/or simulating light reflected by atmosphere-ocean systems becomes impossible. Instead, we focus on both theoretical and experimental studies of RT topics important to the science threshold and goal questions of the PACE mission and the measurement capabilities of its instruments. We differentiate between (a) forward (FWD) RT studies that focus mainly on sensitivity to influencing variables and/or simulating data sets, and (b) inverse (INV) RT studies that also involve the retrieval of atmosphere and ocean parameters. Our topics cover (1) the ocean (i.e., water body): absorption and elastic/inelastic scattering by pure water (FWD RT) and models for scattering and absorption by particulates (FWD RT and INV RT); (2) the air-water interface: variations in ocean surface refractive index (INV RT) and in whitecap reflectance (INV RT); (3) the atmosphere: polarimetric and/or hyperspectral remote sensing of aerosols (INV RT) and of gases (FWD RT); and (4) atmosphere-ocean systems: benchmark comparisons, impact of the Earth’s sphericity and adjacency effects on space-borne observations, and scattering in the ultraviolet regime (FWD RT). We provide for each topic a summary of past relevant (heritage) work, followed by a discussion (for unresolved questions) and RT updates