Sea Ice Extraction via Remote Sensed Imagery: Algorithms, Datasets, Applications and Challenges

The deep learning, which is a dominating technique in artificial intelligence, has completely changed the image understanding over the past decade. As a consequence, the sea ice extraction (SIE) problem has reached a new era. We present a comprehensive review of four important aspects of SIE, including algorithms, datasets, applications, and the future trends. Our review focuses on researches published from 2016 to the present, with a specific focus on deep learning-based approaches in the last five years. We divided all relegated algorithms into 3 categories, including classical image segmentation approach, machine learning-based approach and deep learning-based methods. We reviewed the accessible ice datasets including SAR-based datasets, the optical-based datasets and others. The applications are presented in 4 aspects including climate research, navigation, geographic information systems (GIS) production and others. It also provides insightful observations and inspiring future research directions.Comment: 24 pages, 6 figure

A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote Sensin

Remote Sensing of the Oceans

This book covers different topics in the framework of remote sensing of the oceans. Latest research advancements and brand-new studies are presented that address the exploitation of remote sensing instruments and simulation tools to improve the understanding of ocean processes and enable cutting-edge applications with the aim of preserving the ocean environment and supporting the blue economy. Hence, this book provides a reference framework for state-of-the-art remote sensing methods that deal with the generation of added-value products and the geophysical information retrieval in related fields, including: Oil spill detection and discrimination; Analysis of tropical cyclones and sea echoes; Shoreline and aquaculture area extraction; Monitoring coastal marine litter and moving vessels; Processing of SAR, HF radar and UAV measurements

훈련 자료 자동 추출 알고리즘과 기계 학습을 통한 SAR 영상 기반의 선박 탐지

학위논문 (석사) -- 서울대학교 대학원 : 자연과학대학 지구환경과학부, 2021. 2. 김덕진.Detection and surveillance of vessels are regarded as a crucial application of SAR for their contribution to the preservation of marine resources and the assurance on maritime safety. Introduction of machine learning to vessel detection significantly enhanced the performance and efficiency of the detection, but a substantial majority of studies focused on modifying the object detector algorithm. As the fundamental enhancement of the detection performance would be nearly impossible without accurate training data of vessels, this study implemented AIS information containing real-time information of vessel’s movement in order to propose a robust algorithm which acquires the training data of vessels in an automated manner. As AIS information was irregularly and discretely obtained, the exact target interpolation time for each vessel was precisely determined, followed by the implementation of Kalman filter, which mitigates the measurement error of AIS sensor. In addition, as the velocity of each vessel renders an imprint inside the SAR image named as Doppler frequency shift, it was calibrated by restoring the elliptic satellite orbit from the satellite state vector and estimating the distance between the satellite and the target vessel. From the calibrated position of the AIS sensor inside the corresponding SAR image, training data was directly obtained via internal allocation of the AIS sensor in each vessel. For fishing boats, separate information system named as VPASS was applied for the identical procedure of training data retrieval. Training data of vessels obtained via the automated training data procurement algorithm was evaluated by a conventional object detector, for three detection evaluating parameters: precision, recall and F1 score. All three evaluation parameters from the proposed training data acquisition significantly exceeded that from the manual acquisition. The major difference between two training datasets was demonstrated in the inshore regions and in the vicinity of strong scattering vessels in which land artifacts, ships and the ghost signals derived from them were indiscernible by visual inspection. This study additionally introduced a possibility of resolving the unclassified usage of each vessel by comparing AIS information with the accurate vessel detection results.전천후 지구 관측 위성인 SAR를 통한 선박 탐지는 해양 자원의 확보와 해상 안전 보장에 매우 중요한 역할을 한다. 기계 학습 기법의 도입으로 인해 선박을 비롯한 사물 탐지의 정확도 및 효율성이 향상되었으나, 이와 관련된 다수의 연구는 탐지 알고리즘의 개량에 집중되었다. 그러나, 탐지 정확도의 근본적인 향상은 정밀하게 취득된 대량의 훈련자료 없이는 불가능하기에, 본 연구에서는 선박의 실시간 위치, 속도 정보인 AIS 자료를 이용하여 인공 지능 기반의 선박 탐지 알고리즘에 사용될 훈련자료를 자동적으로 취득하는 알고리즘을 제안하였다. 이를 위해 이산적인 AIS 자료를 SAR 영상의 취득시각에 맞추어 정확하게 보간하고, AIS 센서 자체가 가지는 오차를 최소화하였다. 또한, 이동하는 산란체의 시선 속도로 인해 발생하는 도플러 편이 효과를 보정하기 위해 SAR 위성의 상태 벡터를 이용하여 위성과 산란체 사이의 거리를 정밀하게 계산하였다. 이렇게 계산된 AIS 센서의 영상 내의 위치로부터 선박 내 AIS 센서의 배치를 고려하여 선박 탐지 알고리즘의 훈련자료 형식에 맞추어 훈련자료를 취득하고, 어선에 대한 위치, 속도 정보인 VPASS 자료 역시 유사한 방법으로 가공하여 훈련자료를 취득하였다. AIS 자료로부터 취득한 훈련자료는 기존 방법대로 수동 취득한 훈련자료와 함께 인공 지능 기반 사물 탐지 알고리즘을 통해 정확도를 평가하였다. 그 결과, 제시된 알고리즘으로 취득한 훈련 자료는 수동 취득한 훈련 자료 대비 더 높은 탐지 정확도를 보였으며, 이는 기존의 사물 탐지 알고리즘의 평가 지표인 정밀도, 재현율과 F1 score를 통해 진행되었다. 본 연구에서 제안한 훈련자료 자동 취득 기법으로 얻은 선박에 대한 훈련자료는 특히 기존의 선박 탐지 기법으로는 분별이 어려웠던 항만에 인접한 선박과 산란체 주변의 신호에 대한 정확한 분별 결과를 보였다. 본 연구에서는 이와 함께, 선박 탐지 결과와 해당 지역에 대한 AIS 및 VPASS 자료를 이용하여 선박의 미식별성을 판정할 수 있는 가능성 또한 제시하였다.Chapter 1. Introduction - 1 - 1.1 Research Background - 1 - 1.2 Research Objective - 8 - Chapter 2. Data Acquisition - 10 - 2.1 Acquisition of SAR Image Data - 10 - 2.2 Acquisition of AIS and VPASS Information - 20 - Chapter 3. Methodology on Training Data Procurement - 26 - 3.1 Interpolation of Discrete AIS Data - 29 - 3.1.1 Estimation of Target Interpolation Time for Vessels - 29 - 3.1.2 Application of Kalman Filter to AIS Data - 34 - 3.2 Doppler Frequency Shift Correction - 40 - 3.2.1 Theoretical Basis of Doppler Frequency Shift - 40 - 3.2.2 Mitigation of Doppler Frequency Shift - 48 - 3.3 Retrieval of Training Data of Vessels - 53 - 3.4 Algorithm on Vessel Training Data Acquisition from VPASS Information - 61 - Chapter 4. Methodology on Object Detection Architecture - 66 - Chapter 5. Results - 74 - 5.1 Assessment on Training Data - 74 - 5.2 Assessment on AIS-based Ship Detection - 79 - 5.3 Assessment on VPASS-based Fishing Boat Detection - 91 - Chapter 6. Discussions - 110 - 6.1 Discussion on AIS-Based Ship Detection - 110 - 6.2 Application on Determining Unclassified Vessels - 116 - Chapter 7. Conclusion - 125 - 국문 요약문 - 128 - Bibliography - 130 -Maste

SDF2Net: Shallow to Deep Feature Fusion Network for PolSAR Image Classification

Polarimetric synthetic aperture radar (PolSAR) images encompass valuable information that can facilitate extensive land cover interpretation and generate diverse output products. Extracting meaningful features from PolSAR data poses challenges distinct from those encountered in optical imagery. Deep learning (DL) methods offer effective solutions for overcoming these challenges in PolSAR feature extraction. Convolutional neural networks (CNNs) play a crucial role in capturing PolSAR image characteristics by leveraging kernel capabilities to consider local information and the complex-valued nature of PolSAR data. In this study, a novel three-branch fusion of complex-valued CNN, named the Shallow to Deep Feature Fusion Network (SDF2Net), is proposed for PolSAR image classification. To validate the performance of the proposed method, classification results are compared against multiple state-of-the-art approaches using the airborne synthetic aperture radar (AIRSAR) datasets of Flevoland and San Francisco, as well as the ESAR Oberpfaffenhofen dataset. The results indicate that the proposed approach demonstrates improvements in overallaccuracy, with a 1.3% and 0.8% enhancement for the AIRSAR datasets and a 0.5% improvement for the ESAR dataset. Analyses conducted on the Flevoland data underscore the effectiveness of the SDF2Net model, revealing a promising overall accuracy of 96.01% even with only a 1% sampling ratio

Spatial Modeling of Compact Polarimetric Synthetic Aperture Radar Imagery

The RADARSAT Constellation Mission (RCM) utilizes compact polarimetric (CP) mode to provide data with varying resolutions, supporting a wide range of applications including oil spill detection, sea ice mapping, and land cover analysis. However, the complexity and variability of CP data, influenced by factors such as weather conditions and satellite infrastructure, introduce signature ambiguity. This ambiguity poses challenges in accurate object classification, reducing discriminability and increasing uncertainty. To address these challenges, this thesis introduces tailored spatial models in CP SAR imagery through the utilization of machine learning techniques. Firstly, to enhance oil spill monitoring, a novel conditional random field (CRF) is introduced. The CRF model leverages the statistical properties of CP SAR data and exploits similarities in labels and features among neighboring pixels to effectively model spatial interactions. By mitigating the impact of speckle noise and accurately distinguishing oil spill candidates from oil-free water, the CRF model achieves successful results even in scenarios where the availability of labeled samples is limited. This highlights the capability of CRF in handling situations with a scarcity of training data. Secondly, to improve the accuracy of sea ice mapping, a region-based automated classification methodology is developed. This methodology incorporates learned features, spatial context, and statistical properties from various SAR modes, resulting in enhanced classification accuracy and improved algorithmic efficiency. Thirdly, the presence of a high degree of heterogeneity in target distribution presents an additional challenge in land cover mapping tasks, further compounded by signature ambiguity. To address this, a novel transformer model is proposed. The transformer model incorporates both fine- and coarse-grained spatial dependencies between pixels and leverages different levels of features to enhance the accuracy of land cover type detection. The proposed approaches have undergone extensive experimentation in various remote sensing tasks, validating their effectiveness. By introducing tailored spatial models and innovative algorithms, this thesis successfully addresses the inherent complexity and variability of CP data, thereby ensuring the accuracy and reliability of diverse applications in the field of remote sensing

Space-based Global Maritime Surveillance. Part I: Satellite Technologies

Maritime surveillance (MS) is crucial for search and rescue operations, fishery monitoring, pollution control, law enforcement, migration monitoring, and national security policies. Since the early days of seafaring, MS has been a critical task for providing security in human coexistence. Several generations of sensors providing detailed maritime information have become available for large offshore areas in real time: maritime radar sensors in the 1950s and the automatic identification system (AIS) in the 1990s among them. However, ground-based maritime radars and AIS data do not always provide a comprehensive and seamless coverage of the entire maritime space. Therefore, the exploitation of space-based sensor technologies installed on satellites orbiting around the Earth, such as satellite AIS data, synthetic aperture radar, optical sensors, and global navigation satellite systems reflectometry, becomes crucial for MS and to complement the existing terrestrial technologies. In the first part of this work, we provide an overview of the main available space-based sensors technologies and present the advantages and limitations of each technology in the scope of MS. The second part, related to artificial intelligence, signal processing and data fusion techniques, is provided in a companion paper, titled: "Space-based Global Maritime Surveillance. Part II: Artificial Intelligence and Data Fusion Techniques" [1].Comment: This paper has been submitted to IEEE Aerospace and Electronic Systems Magazin

Analyzing Depthwise Convolution Based Neural Network: Study Case in Ship Detection and Land Cover Classification

Various methods are available to perform feature extraction on satellite images. Among the available alternatives, deep convolutional neural network (ConvNet) is the state of the art method. Although previous studies have reported successful attempts on developing and implementing ConvNet on remote sensing application, several issues are not well explored, such as the use of depthwise convolution, final pooling layer size, and comparison between grayscale and RGB settings. The objective of this study is to perform analysis to address these issues. Two feature learning algorithms were proposed, namely ConvNet as the current state of the art for satellite image classification and Gray Level Co-occurence Matrix (GLCM) which represents a classic unsupervised feature extraction method. The experiment demonstrated consistent result with previous studies that ConvNet is superior in most cases compared to GLCM, especially with 3x3xn final pooling. The performance of the learning algorithms are much higher on features from RGB channels, except for ConvNet with relatively small number of features

Synthetic Aperture Radar (SAR) Meets Deep Learning

This reprint focuses on the application of the combination of synthetic aperture radars and depth learning technology. It aims to further promote the development of SAR image intelligent interpretation technology. A synthetic aperture radar (SAR) is an important active microwave imaging sensor, whose all-day and all-weather working capacity give it an important place in the remote sensing community. Since the United States launched the first SAR satellite, SAR has received much attention in the remote sensing community, e.g., in geological exploration, topographic mapping, disaster forecast, and traffic monitoring. It is valuable and meaningful, therefore, to study SAR-based remote sensing applications. In recent years, deep learning represented by convolution neural networks has promoted significant progress in the computer vision community, e.g., in face recognition, the driverless field and Internet of things (IoT). Deep learning can enable computational models with multiple processing layers to learn data representations with multiple-level abstractions. This can greatly improve the performance of various applications. This reprint provides a platform for researchers to handle the above significant challenges and present their innovative and cutting-edge research results when applying deep learning to SAR in various manuscript types, e.g., articles, letters, reviews and technical reports