A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote Sensin

Two-Phase Object-Based Deep Learning for Multi-Temporal SAR Image Change Detection

Change detection is one of the fundamental applications of synthetic aperture radar (SAR) images. However, speckle noise presented in SAR images has a negative effect on change detection, leading to frequent false alarms in the mapping products. In this research, a novel two-phase object-based deep learning approach is proposed for multi-temporal SAR image change detection. Compared with traditional methods, the proposed approach brings two main innovations. One is to classify all pixels into three categories rather than two categories: unchanged pixels, changed pixels caused by strong speckle (false changes), and changed pixels formed by real terrain variation (real changes). The other is to group neighbouring pixels into superpixel objects such as to exploit local spatial context. Two phases are designed in the methodology: (1) Generate objects based on the simple linear iterative clustering (SLIC) algorithm, and discriminate these objects into changed and unchanged classes using fuzzy c-means (FCM) clustering and a deep PCANet. The prediction of this Phase is the set of changed and unchanged superpixels. (2) Deep learning on the pixel sets over the changed superpixels only, obtained in the first phase, to discriminate real changes from false changes. SLIC is employed again to achieve new superpixels in the second phase. Low rank and sparse decomposition are applied to these new superpixels to suppress speckle noise significantly. A further clustering step is applied to these new superpixels via FCM. A new PCANet is then trained to classify two kinds of changed superpixels to achieve the final change maps. Numerical experiments demonstrate that, compared with benchmark methods, the proposed approach can distinguish real changes from false changes effectively with significantly reduced false alarm rates, and achieve up to 99.71% change detection accuracy using multi-temporal SAR imagery

A self-paced learning algorithm for change detection in synthetic aperture radar images

Detecting changed regions between two given synthetic aperture radar images is very important to monitor the change of landscapes, change of ecosystem and so on. This can be formulated as a classification problem and addressed by learning a classifier, traditional machine learning classification methods very easily stick to local optima which can be caused by noises of data. Hence, we propose an unsupervised algorithm aiming at constructing a classifier based on self-paced learning. Self-paced learning is a recently developed supervised learning approach and has been proven to be capable to overcome effectively this shortcoming. After applying a pre-classification to the difference image, we uniformly select samples using the initial result. Then, self-paced learning is utilized to train a classifier. Finally, a filter is used based on spatial contextual information to further smooth the classification result. In order to demonstrate the efficiency of the proposed algorithm, we apply our proposed algorithm on five real synthetic aperture radar images datasets. The results obtained by our algorithm are compared with five other state-of-the-art algorithms, which demonstrates that our algorithm outperforms those state-of-the-art algorithms in terms of accuracy and robustness

Autonomous real-time infrared detection of sub-surface vessels for unmanned aircraft systems

The threat of small self-propelled semi-submersible vessels cannot be understated; payloads from drugs to weapons of mass destruction could be housed in these small, inconspicuous vessels. With a current apprehension rate of approximately 10%, a method resulting in increased interdiction of this illegal traffic is required for national security both in the ports along the coastlines of Canada, as well as the rest of North America. A smart, autonomous payload containing an infrared imaging device, designed for use in small unmanned aircraft systems for the specific mission of detecting self-propelled semi-submersibles over the vast ocean coastline will address the current security needs. Thermal imagery of the disturbed colder water layers, driven to the surface by the vessel will allow for the detection of this traffic using long wave infrared technology. Infrared signatures of ship wakes are highly variable in both persistence and temperature contrast as compared to the surrounding surface water, thus infrared imaging devices with a high resolution, a high responsivity, and a very low minimum resolvable temperature will be required to provide high quality imagery for airborne detection of the thermal wake. A theoretical understanding of the physics associated with the energy collected by the infrared sensor and the resulting infrared images is provided. Explanation of the factors affecting the resulting image with respect to the camera properties are detailed. A variety of examples of airborne thermal images are presented, with detailed explanations of the imaged scenes based on theory and sensor characteristics provided in the previous sections. Infrared images taken over the Atlantic and Pacific oceans from manned and unmanned aircraft platforms are presented. Temperature measurements taken using Vemco Minilog II temperature loggers confirmed the thermal stratification of the upper 5 meters of the water. Thermal scarring due to upwelled colder water to the surface was noted during the day time under normal conditions, with temperature differences found to be consistent with the measured temperature profile. A custom gimbal system, with corresponding ground control station for real-time, visual feedback is presented. An algorithm for the detection of submerged vessel ship wakes using a LWIR camera, specifically for a small unmanned aircraft, with limited power, space, and computing power is developed. A time sequential processing method is presented to reduce the required computing, while allowing high frame rate, real-time operation. Moreover, a windowed triple-vote method is continually applied to ensure that the detection mode is correctly set by the algorithm, while ignoring unexpected targets in the image. A simple background estimation method is presented to remove any nonuniformity in the captured images, resulting in a high detection rate with low false alarms. Finally, a complete, mission-ready payload system is prepared for small UA platforms, with an accuracy rate greater than 97% for the detection of self-propelled semi-submersible vessels