A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote Sensin

Deep learning in remote sensing: a review

Standing at the paradigm shift towards data-intensive science, machine learning techniques are becoming increasingly important. In particular, as a major breakthrough in the field, deep learning has proven as an extremely powerful tool in many fields. Shall we embrace deep learning as the key to all? Or, should we resist a 'black-box' solution? There are controversial opinions in the remote sensing community. In this article, we analyze the challenges of using deep learning for remote sensing data analysis, review the recent advances, and provide resources to make deep learning in remote sensing ridiculously simple to start with. More importantly, we advocate remote sensing scientists to bring their expertise into deep learning, and use it as an implicit general model to tackle unprecedented large-scale influential challenges, such as climate change and urbanization.Comment: Accepted for publication IEEE Geoscience and Remote Sensing Magazin

End-to-end Trainable Ship Detection in SAR Images with Single Level Features

Kongsberg Satellite Services (KSAT) use machine learning and manual analysis done by synthetic aperture radar (SAR) specialists on SAR images in real time to provide a ship detection service. KSATs current machine learning model has a limited ability to distinguish ships close to each other. For this reason, we aim to employ an end-to-end trainable object detection model, as they can better distinguish nearby objects, since they are not limited by heuristic post processing. Since heuristic post processing in object detection limit the models ability to distinguish ships close to each other, we investigate challenges related to employing an end-to-end trainable ship detection model. Since access to ground truth annotations in SAR images is limited, size and rotation labels are not available for all ships, and rotation labels are inaccurate. Since KSATs internal datasets are collected as part of a time critical operational service, position labels are not exact. Since existing evaluation metrics for object detection are too strict, they do not reflect user needs for this service. To tolerate missing size and rotation annotations, we base loss label assignment on the distance between objects instead of their IoU, and replace DIoU bounding box loss with a novel size regression loss named Size IoU (SIoU) combined with smooth L1 position loss. To tolerate inaccurate rotation labels, we propose angular direction vector (ADV) regression. To tolerate inaccurate position labels, the loss label assignment makes all predictions responsible for large overlapping regions instead of small disjoint regions. To compare models performance according to user needs, we propose an evaluation metric named Distance-AP (dAP), which is based on mAP, but replaces the IoU overlap threshold with an object center point distance threshold. To reduce duplicate ship predictions, we propose multi layer attention. Using the LS-SSDD SAR ship dataset, we find that replacing IoU based label assignment with position based label assignment increases dAP from 79% to 86%, and that replacing DIoU with SIoU decreases dAP by only 1%. Using a rotation regression benchmark where datasets have different amounts of rotation label noise, we find that ADV outperforms CSL in terms of mean predicted inaccuracy at all noise levels, and median predicted inaccuracy at high noise levels. Using an object detection benchmark where the datasets have varying amount of position label inaccuracy, we find that the proposed loss label assignment tolerates large amounts of noise without reduced performance. Using KSATs dataset of Sentinel 1 images, we measure 83% dAP. The proposed mechanisms allow effective training of a ship detection model, despite the missing size and rotation annotations, inaccurate position annotations, and inaccurate rotation annotations. We believe this is useful for KSATs ship detection service, as it can better distinguish nearby ships. However, more work is required to compare its performance with their existing solution. Source code is available at https://github.com/matill/Ship-detectio

Automated High-resolution Earth Observation Image Interpretation: Outcome of the 2020 Gaofen Challenge

In this article, we introduce the 2020 Gaofen Challenge and relevant scientific outcomes. The 2020 Gaofen Challenge is an international competition, which is organized by the China High-Resolution Earth Observation Conference Committee and the Aerospace Information Research Institute, Chinese Academy of Sciences and technically cosponsored by the IEEE Geoscience and Remote Sensing Society and the International Society for Photogrammetry and Remote Sensing. It aims at promoting the academic development of automated high-resolution earth observation image interpretation. Six independent tracks have been organized in this challenge, which cover the challenging problems in the field of object detection and semantic segmentation. With the development of convolutional neural networks, deep-learning-based methods have achieved good performance on image interpretation. In this article, we report the details and the best-performing methods presented so far in the scope of this challenge

Offshore oil spill detection using synthetic aperture radar

Among the different types of marine pollution, oil spill has been considered as a major threat to the sea ecosystems. The source of the oil pollution can be located on the mainland or directly at sea. The sources of oil pollution at sea are discharges coming from ships, offshore platforms or natural seepage from sea bed. Oil pollution from sea-based sources can be accidental or deliberate. Different sensors to detect and monitor oil spills could be onboard vessels, aircraft, or satellites. Vessels equipped with specialised radars, can detect oil at sea but they can cover a very limited area. One of the established ways to monitor sea-based oil pollution is the use of satellites equipped with Synthetic Aperture Radar (SAR).The aim of the work presented in this thesis is to identify optimum set of feature extracted parameters and implement methods at various stages for oil spill detection from Synthetic Aperture Radar (SAR) imagery. More than 200 images of ERS-2, ENVSAT and RADARSAT 2 SAR sensor have been used to assess proposed feature vector for oil spill detection methodology, which involves three stages: segmentation for dark spot detection, feature extraction and classification of feature vector. Unfortunately oil spill is not only the phenomenon that can create a dark spot in SAR imagery. There are several others meteorological and oceanographic and wind induced phenomena which may lead to a dark spot in SAR imagery. Therefore, these dark objects also appear similar to the dark spot due to oil spill and are called as look-alikes. These look-alikes thus cause difficulty in detecting oil spill spots as their primary characteristic similar to oil spill spots. To get over this difficulty, feature extraction becomes important; a stage which may involve selection of appropriate feature extraction parameters. The main objective of this dissertation is to identify the optimum feature vector in order to segregate oil spill and ‘look-alike’ spots. A total of 44 Feature extracted parameters have been studied. For segmentation, four methods; based on edge detection, adaptive theresholding, artificial neural network (ANN) segmentation and the other on contrast split segmentation have been implemented. Spot features are extracted from both the dark spots themselves and their surroundings. Classification stage was performed using two different classification techniques, first one is based on ANN and the other based on a two-stage processing that combines classification tree analysis and fuzzy logic. A modified feature vector, including both new and improved features, is suggested for better description of different types of dark spots. An ANN classifier using full spectrum of feature parameters has also been developed and evaluated. The implemented methodology appears promising in detecting dark spots and discriminating oil spills from look-alikes and processing time is well below any operational service requirements

Research on Ship Classification using Faster Region Convolutional Neural Network for Port Security

Huvudsyftet med studien var att se i vilken grad det går att finna samarbeten genom material- och/eller energiutbyten mellan närliggande anläggningar inom skogsindustrin i Sverige. Genom att göra en inventering av vilka anläggningar som finns inom skogsindustrin och sedan kontakta dessa, sammanställdes en lista över de olika anläggningarna och deras olika samarbeten. Inventeringen gjordes med hjälp av olika branschorganisationer samt sökmotorer på Internet. Utöver detta besöktes också fyra intressanta fall för att ge en inblick i hur dessa samarbeten kan se ut. Studien visar på att den här typen av samarbeten existerar inom skogsindustrin och att drygt en tredjedel av de studerade anläggningarna har någon form av samarbeten rörande dessa frågor. Detta pekar på att man inom skogsindustrin är långt framme när det gäller resursutnyttjande och att möjligheten att minimera sin energi- och materialanvändning hela tiden är en relevant fråga. Det finns med stor sannolikhet ännu fler sådana samarbeten som inte framkommit vid undersökningen och en intressant aspekt är att vid de besök som gjordes upptäcktes samarbeten som inte uppmärksammats vid tidigare kontakter. Av de 152 tillfrågade anläggningarna i inventeringen erhölls svar från 117 stycken vilket tyder på att det finns ett stort intresse för dessa frågor inom skogsindustrin. Flera av de anläggningar som inte hade några samarbeten kring dessa frågor svarade också att de hela tiden undersöker möjligheten till att inleda sådana. Många av samarbetena rörande dessa frågor kretsar kring leveranser av el och ånga samt spån och flis men en del andra intressanta samarbeten har också framkommit. Exempelvis används slam från bioreningsdammar till bränsle, jordförbättringsmedel och som täckmaterial vid deponier. Sammanfattningsvis tyder detta på att skogsindustrin ligger långt framme gällande dessa frågor men att det fortfarande finns mer att göra om energi- och materialanvändningen och därigenom den negativa miljöpåverkan ska minimeras.The aim and objective with this study was to investigate to what extent co-operation through material and energy exchange between adjacent industries among the forest industry in Sweden could be found. First, an inventory of the industries in the forest industry was conducted. Secondly, each company was contacted with questions concerning this issue. Complementary field studies of four specific cases were conducted in order to give an insight to how these co-operations may function in reality. The result of this study illustrates that co-operations among the industries exist in the forest industry sector as more than a third of the investigated industries has some kind of co-operation regarding material and energy exchange with adjacent industries. A total number of 152 industries were identified during the inventory phase and 117 of those industries participated in the study with their own answers. This high participation rate enhances the impression that these are important questions to the forest industry sector. Numerous of the co-operations mentioned revolve around electricity, steam, and by products from sawmills, like woodchips and sawdust. Nevertheless, a few other interesting co-operations have also been revealed during the study, for example; sludge from some of the pulp mills are used as fuel, soil fertilizer and as covering material at landfills. An interesting point is that co-operations, which not were discovered during the earlier correspondence with the industries, in fact were revealed during the field studies. Therefore, the probability that there are more existing co-operations between adjacent industries than the findings in the study reveals, are high. To sum up, this shows that the forest industry is well in advance regarding co-operation through material and energy exchange between adjacent industries. However, there is still a lot to be done if the negative effect on the environment from the forest industry should be minimised

Synthetic Aperture Radar (SAR) Meets Deep Learning

This reprint focuses on the application of the combination of synthetic aperture radars and depth learning technology. It aims to further promote the development of SAR image intelligent interpretation technology. A synthetic aperture radar (SAR) is an important active microwave imaging sensor, whose all-day and all-weather working capacity give it an important place in the remote sensing community. Since the United States launched the first SAR satellite, SAR has received much attention in the remote sensing community, e.g., in geological exploration, topographic mapping, disaster forecast, and traffic monitoring. It is valuable and meaningful, therefore, to study SAR-based remote sensing applications. In recent years, deep learning represented by convolution neural networks has promoted significant progress in the computer vision community, e.g., in face recognition, the driverless field and Internet of things (IoT). Deep learning can enable computational models with multiple processing layers to learn data representations with multiple-level abstractions. This can greatly improve the performance of various applications. This reprint provides a platform for researchers to handle the above significant challenges and present their innovative and cutting-edge research results when applying deep learning to SAR in various manuscript types, e.g., articles, letters, reviews and technical reports