Recent Advances in Image Restoration with Applications to Real World Problems

In the past few decades, imaging hardware has improved tremendously in terms of resolution, making widespread usage of images in many diverse applications on Earth and planetary missions. However, practical issues associated with image acquisition are still affecting image quality. Some of these issues such as blurring, measurement noise, mosaicing artifacts, low spatial or spectral resolution, etc. can seriously affect the accuracy of the aforementioned applications. This book intends to provide the reader with a glimpse of the latest developments and recent advances in image restoration, which includes image super-resolution, image fusion to enhance spatial, spectral resolution, and temporal resolutions, and the generation of synthetic images using deep learning techniques. Some practical applications are also included

Spatially Enhanced Spectral Unmixing Through Data Fusion of Spectral and Visible Images from Different Sensors

Publiher's version (útgefin grein)We propose an unmixing framework for enhancing endmember fraction maps using a combination of spectral and visible images. The new method, data fusion through spatial information-aided learning (DFuSIAL), is based on a learning process for the fusion of a multispectral image of low spatial resolution and a visible RGB image of high spatial resolution. Unlike commonly used methods, DFuSIAL allows for fusing data from different sensors. To achieve this objective, we apply a learning process using automatically extracted invariant points, which are assumed to have the same land cover type in both images. First, we estimate the fraction maps of a set of endmembers for the spectral image. Then, we train a spatial-features aided neural network (SFFAN) to learn the relationship between the fractions, the visible bands, and rotation-invariant spatial features for learning (RISFLs) that we extract from the RGB image. Our experiments show that the proposed DFuSIAL method obtains fraction maps with significantly enhanced spatial resolution and an average mean absolute error between 2% and 4% compared to the reference ground truth. Furthermore, it is shown that the proposed method is preferable to other examined state-of-the-art methods, especially when data is obtained from different instruments and in cases with missing-data pixels.This research was partially funded by the Icelandic Research Fund through the EMMIRS project, and bythe Israel Science Ministry and Space Agency through the Venus project.Peer Reviewe

Deep learning in remote sensing: a review

Standing at the paradigm shift towards data-intensive science, machine learning techniques are becoming increasingly important. In particular, as a major breakthrough in the field, deep learning has proven as an extremely powerful tool in many fields. Shall we embrace deep learning as the key to all? Or, should we resist a 'black-box' solution? There are controversial opinions in the remote sensing community. In this article, we analyze the challenges of using deep learning for remote sensing data analysis, review the recent advances, and provide resources to make deep learning in remote sensing ridiculously simple to start with. More importantly, we advocate remote sensing scientists to bring their expertise into deep learning, and use it as an implicit general model to tackle unprecedented large-scale influential challenges, such as climate change and urbanization.Comment: Accepted for publication IEEE Geoscience and Remote Sensing Magazin

Development of an unsupervised remote sensing methodology of detect surface leakage from terrestrial CO2 storage sites

Imperial Users onl

Multisource and Multitemporal Data Fusion in Remote Sensing

The sharp and recent increase in the availability of data captured by different sensors combined with their considerably heterogeneous natures poses a serious challenge for the effective and efficient processing of remotely sensed data. Such an increase in remote sensing and ancillary datasets, however, opens up the possibility of utilizing multimodal datasets in a joint manner to further improve the performance of the processing approaches with respect to the application at hand. Multisource data fusion has, therefore, received enormous attention from researchers worldwide for a wide variety of applications. Moreover, thanks to the revisit capability of several spaceborne sensors, the integration of the temporal information with the spatial and/or spectral/backscattering information of the remotely sensed data is possible and helps to move from a representation of 2D/3D data to 4D data structures, where the time variable adds new information as well as challenges for the information extraction algorithms. There are a huge number of research works dedicated to multisource and multitemporal data fusion, but the methods for the fusion of different modalities have expanded in different paths according to each research community. This paper brings together the advances of multisource and multitemporal data fusion approaches with respect to different research communities and provides a thorough and discipline-specific starting point for researchers at different levels (i.e., students, researchers, and senior researchers) willing to conduct novel investigations on this challenging topic by supplying sufficient detail and references

A Deep Learning Framework in Selected Remote Sensing Applications

The main research topic is designing and implementing a deep learning framework applied to remote sensing. Remote sensing techniques and applications play a crucial role in observing the Earth evolution, especially nowadays, where the effects of climate change on our life is more and more evident. A considerable amount of data are daily acquired all over the Earth. Effective exploitation of this information requires the robustness, velocity and accuracy of deep learning. This emerging need inspired the choice of this topic. The conducted studies mainly focus on two European Space Agency (ESA) missions: Sentinel 1 and Sentinel 2. Images provided by the ESA Sentinel-2 mission are rapidly becoming the main source of information for the entire remote sensing community, thanks to their unprecedented combination of spatial, spectral and temporal resolution, as well as their open access policy. The increasing interest gained by these satellites in the research laboratory and applicative scenarios pushed us to utilize them in the considered framework. The combined use of Sentinel 1 and Sentinel 2 is crucial and very prominent in different contexts and different kinds of monitoring when the growing (or changing) dynamics are very rapid. Starting from this general framework, two specific research activities were identified and investigated, leading to the results presented in this dissertation. Both these studies can be placed in the context of data fusion. The first activity deals with a super-resolution framework to improve Sentinel 2 bands supplied at 20 meters up to 10 meters. Increasing the spatial resolution of these bands is of great interest in many remote sensing applications, particularly in monitoring vegetation, rivers, forests, and so on. The second topic of the deep learning framework has been applied to the multispectral Normalized Difference Vegetation Index (NDVI) extraction, and the semantic segmentation obtained fusing Sentinel 1 and S2 data. The S1 SAR data is of great importance for the quantity of information extracted in the context of monitoring wetlands, rivers and forests, and many other contexts. In both cases, the problem was addressed with deep learning techniques, and in both cases, very lean architectures were used, demonstrating that even without the availability of computing power, it is possible to obtain high-level results. The core of this framework is a Convolutional Neural Network (CNN). {CNNs have been successfully applied to many image processing problems, like super-resolution, pansharpening, classification, and others, because of several advantages such as (i) the capability to approximate complex non-linear functions, (ii) the ease of training that allows to avoid time-consuming handcraft filter design, (iii) the parallel computational architecture. Even if a large amount of "labelled" data is required for training, the CNN performances pushed me to this architectural choice.} In our S1 and S2 integration task, we have faced and overcome the problem of manually labelled data with an approach based on integrating these two different sensors. Therefore, apart from the investigation in Sentinel-1 and Sentinel-2 integration, the main contribution in both cases of these works is, in particular, the possibility of designing a CNN-based solution that can be distinguished by its lightness from a computational point of view and consequent substantial saving of time compared to more complex deep learning state-of-the-art solutions

시공간 해상도 향상을 통한 식생 변화 모니터링

학위논문(박사) -- 서울대학교대학원 : 환경대학원 협동과정 조경학, 2023. 2. 류영렬.육상 생태계에서 대기권과 생물권의 상호 작용을 이해하기 위해서는 식생 변화의 모니터링이 필요하다. 이 때, 위성영상은 지표면을 관측하여 식생지도를 제공할 수 있지만, 지표변화의 상세한 정보는 구름이나 위성 이미지의 공간 해상도에 의해 제한되었다. 또한 위성영상의 시공간 해상도가 식생지도를 통한 광합성 모니터링에 미치는 영향은 완전히 밝혀지지 않았다. 본 논문에서는 고해상도 식생 지도를 일단위로 생성하기 위성 영상의 시공간 해상도를 향상시키는 것을 목표로 하였다. 고해상도 위성영상을 활용한 식생 변화 모니터링을 시공간적으로 확장하기 위해 1) 정지궤도 위성을 활용한 영상융합을 통해 시간해상도 향상, 2) 적대적생성네트워크를 활용한 공간해상도 향상, 3) 시공간해상도가 높은 위성영상을 토지피복이 균질하지 않은 공간에서 식물 광합성 모니터링을 수행하였다. 이처럼, 위성기반 원격탐지에서 새로운 기술이 등장함에 따라 현재 및 과거의 위성영상은 시공간 해상도 측면에서 향상되어 식생 변화의 모니터링 할 수 있다. 제2장에서는 정지궤도위성영상을 활용하는 시공간 영상융합으로 식물의 광합성을 모니터링 했을 때, 시간해상도가 향상됨을 보였다. 시공간 영상융합 시, 구름탐지, 양방향 반사 함수 조정, 공간 등록, 시공간 융합, 시공간 결측치 보완 등의 과정을 거친다. 이 영상융합 산출물은 경작관리 등으로 식생 지수의 연간 변동이 큰 두 장소(농경지와 낙엽수림)에서 평가하였다. 그 결과, 시공간 영상융합 산출물은 결측치 없이 현장관측을 예측하였다 (R2 = 0.71, 상대 편향 = 5.64% 농경지; R2 = 0.79, 상대 편향 = -13.8%, 활엽수림). 시공간 영상융합은 식생 지도의 시공간 해상도를 점진적으로 개선하여, 식물 생장기동안 위성영상이 현장 관측을 과소 평가를 줄였다. 영상융합은 높은 시공간 해상도로 광합성 지도를 일간격으로 생성하기에 이를 활용하여 위성 영상의 제한된 시공간 해상도로 밝혀지지 않은 식물변화의 과정을 발견하길 기대한다. 식생의 공간분포은 정밀농업과 토지 피복 변화 모니터링을 위해 필수적이다. 고해상도 위성영상으로 지구 표면을 관측하는 것을 용이하게 해졌다. 특히 Planet Fusion은 초소형위성군 데이터를 최대한 활용해 데이터 결측이 없는 3m 공간 해상도의 지표 표면 반사도이다. 그러나 과거 위성 센서(Landsat의 경우 30~60m)의 공간 해상도는 식생의 공간적 변화를 상세 분석하는 것을 제한했다. 제3장에서는 Landsat 데이터의 공간 해상도를 향상하기 위해 Planet Fusion 및 Landsat 8 데이터를 사용하여 이중 적대적 생성 네트워크(the dual RSS-GAN)를 학습시켜, 고해상도 정규화 식생 지수(NDVI)와 식물 근적외선 반사(NIRv)도를 생성하는 한다. 타워기반 현장 식생지수(최대 8년)와 드론기반 초분광지도로 the dual RSS-GAN의 성능을 대한민국 내 두 대상지(농경지와 활엽수림)에서 평가했다. The dual RSS-GAN은 Landsat 8 영상의 공간해상도를 향상시켜 공간 표현을 보완하고 식생 지수의 계절적 변화를 포착했다(R2> 0.96). 그리고 the dual RSS-GAN은 Landsat 8 식생 지수가 현장에 비해 과소 평가되는 것을 완화했다. 현장 관측에 비해 이중 RSS-GAN과 Landsat 8의 상대 편향 값 각각 -0.8% 에서 -1.5%, -10.3% 에서 -4.6% 였다. 이러한 개선은 Planet Fusion의 공간정보를 이중 RSS-GAN로 학습하였기에 가능했다. 헤당 연구 결과는 Landsat 영상의 공간 해상도를 향상시켜 숨겨진 공간 정보를 제공하는 새로운 접근 방식이다. 고해상도에서 식물 광합성 지도는 토지피복이 복잡한 공간에서 탄소 순환 모니터링시 필수적이다. 그러나 Sentinel-2, Landsat 및 MODIS와 같이 태양 동조 궤도에 있는 위성은 공간 해상도가 높거나 시간 해상도 높은 위성영상만 제공할 수 있다. 최근 발사된 초소형위성군은 이러한 해상도 한계을 극복할 수 있다. 특히 Planet Fusion은 초소형위성 자료의 시공간 해상도로 지표면을 관측할 수 있다. 4장에서, Planet Fusion 지표반사도를 이용하여 식생에서 반사된 근적외선 복사(NIRvP)를 3m 해상도 지도를 일간격으로 생성했다. 그런 다음 미국 캘리포니아주 새크라멘토-샌 호아킨 델타의 플럭스 타워 네트워크 데이터와 비교하여 식물 광합성을 추정하기 위한 NIRvP 지도의 성능을 평가하였다. 전체적으로 NIRvP 지도는 습지의 잦은 수위 변화에도 불구하고 개별 대상지의 식물 광합성의 시간적 변화를 포착하였다. 그러나 대상지 전체에 대한 NIRvP 지도와 식물 광합성 사이의 관계는 NIRvP 지도를 플럭스 타워 관측범위와 일치시킬 때만 높은 상관관계를 보였다. 관측범위를 일치시킬 경우, NIRvP 지도는 식물 광합성을 추정하는 데 있어 현장 NIRvP보다 우수한 성능을 보였다. 이러한 성능 차이는 플럭스 타워 관측범위를 일치시킬 때, 연구 대상지 간의 NIRvP-식물 광합성 관계의 기울기가 일관성을 보였기 때문이다. 본 연구 결과는 위성 관측을 플럭스 타워 관측범위와 일치시키는 것의 중요성을 보여주고 높은 시공간 해상도로 식물 광합성을 원격으로 모니터링하는 초소형위성군 자료의 잠재력을 보여준다.Monitoring changes in terrestrial vegetation is essential to understanding interactions between atmosphere and biosphere, especially terrestrial ecosystem. To this end, satellite remote sensing offer maps for examining land surface in different scales. However, the detailed information was hindered under the clouds or limited by the spatial resolution of satellite imagery. Moreover, the impacts of spatial and temporal resolution in photosynthesis monitoring were not fully revealed. In this dissertation, I aimed to enhance the spatial and temporal resolution of satellite imagery towards daily gap-free vegetation maps with high spatial resolution. In order to expand vegetation change monitoring in time and space using high-resolution satellite images, I 1) improved temporal resolution of satellite dataset through image fusion using geostationary satellites, 2) improved spatial resolution of satellite dataset using generative adversarial networks, and 3) showed the use of high spatiotemporal resolution maps for monitoring plant photosynthesis especially over heterogeneous landscapes. With the advent of new techniques in satellite remote sensing, current and past datasets can be fully utilized for monitoring vegetation changes in the respect of spatial and temporal resolution. In Chapter 2, I developed the integrated system that implemented geostationary satellite products in the spatiotemporal image fusion method for monitoring canopy photosynthesis. The integrated system contains the series of process (i.e., cloud masking, nadir bidirectional reflectance function adjustment, spatial registration, spatiotemporal image fusion, spatial gap-filling, temporal-gap-filling). I conducted the evaluation of the integrated system over heterogeneous rice paddy landscape where the drastic land cover changes were caused by cultivation management and deciduous forest where consecutive changes occurred in time. The results showed that the integrated system well predict in situ measurements without data gaps (R2 = 0.71, relative bias = 5.64% at rice paddy site; R2 = 0.79, relative bias = -13.8% at deciduous forest site). The integrated system gradually improved the spatiotemporal resolution of vegetation maps, reducing the underestimation of in situ measurements, especially during peak growing season. Since the integrated system generates daily canopy photosynthesis maps for monitoring dynamics among regions of interest worldwide with high spatial resolution. I anticipate future efforts to reveal the hindered information by the limited spatial and temporal resolution of satellite imagery. Detailed spatial representations of terrestrial vegetation are essential for precision agricultural applications and the monitoring of land cover changes in heterogeneous landscapes. The advent of satellite-based remote sensing has facilitated daily observations of the Earths surface with high spatial resolution. In particular, a data fusion product such as Planet Fusion has realized the delivery of daily, gap-free surface reflectance data with 3-m pixel resolution through full utilization of relatively recent (i.e., 2018-) CubeSat constellation data. However, the spatial resolution of past satellite sensors (i.e., 30–60 m for Landsat) has restricted the detailed spatial analysis of past changes in vegetation. In Chapter 3, to overcome the spatial resolution constraint of Landsat data for long-term vegetation monitoring, we propose a dual remote-sensing super-resolution generative adversarial network (dual RSS-GAN) combining Planet Fusion and Landsat 8 data to simulate spatially enhanced long-term time-series of the normalized difference vegetation index (NDVI) and near-infrared reflectance from vegetation (NIRv). We evaluated the performance of the dual RSS-GAN against in situ tower-based continuous measurements (up to 8 years) and remotely piloted aerial system-based maps of cropland and deciduous forest in the Republic of Korea. The dual RSS-GAN enhanced spatial representations in Landsat 8 images and captured seasonal variation in vegetation indices (R2 > 0.95, for the dual RSS-GAN maps vs. in situ data from all sites). Overall, the dual RSS-GAN reduced Landsat 8 vegetation index underestimations compared with in situ measurements; relative bias values of NDVI ranged from −3.2% to 1.2% and −12.4% to −3.7% for the dual RSS-GAN and Landsat 8, respectively. This improvement was caused by spatial enhancement through the dual RSS-GAN, which captured fine-scale information from Planet Fusion. This study presents a new approach for the restoration of hidden sub-pixel spatial information in Landsat images. Mapping canopy photosynthesis in both high spatial and temporal resolution is essential for carbon cycle monitoring in heterogeneous areas. However, well established satellites in sun-synchronous orbits such as Sentinel-2, Landsat and MODIS can only provide either high spatial or high temporal resolution but not both. Recently established CubeSat satellite constellations have created an opportunity to overcome this resolution trade-off. In particular, Planet Fusion allows full utilization of the CubeSat data resolution and coverage while maintaining high radiometric quality. In Chapter 4, I used the Planet Fusion surface reflectance product to calculate daily, 3-m resolution, gap-free maps of the near-infrared radiation reflected from vegetation (NIRvP). I then evaluated the performance of these NIRvP maps for estimating canopy photosynthesis by comparing with data from a flux tower network in Sacramento-San Joaquin Delta, California, USA. Overall, NIRvP maps captured temporal variations in canopy photosynthesis of individual sites, despite changes in water extent in the wetlands and frequent mowing in the crop fields. When combining data from all sites, however, I found that robust agreement between NIRvP maps and canopy photosynthesis could only be achieved when matching NIRvP maps to the flux tower footprints. In this case of matched footprints, NIRvP maps showed considerably better performance than in situ NIRvP in estimating canopy photosynthesis both for daily sum and data around the time of satellite overpass (R2 = 0.78 vs. 0.60, for maps vs. in situ for the satellite overpass time case). This difference in performance was mostly due to the higher degree of consistency in slopes of NIRvP-canopy photosynthesis relationships across the study sites for flux tower footprint-matched maps. Our results show the importance of matching satellite observations to the flux tower footprint and demonstrate the potential of CubeSat constellation imagery to monitor canopy photosynthesis remotely at high spatio-temporal resolution.Chapter 1. Introduction 2 1. Background 2 1.1 Daily gap-free surface reflectance using geostationary satellite products 2 1.2 Monitoring past vegetation changes with high-spatial-resolution 3 1.3 High spatiotemporal resolution vegetation photosynthesis maps 4 2. Purpose of Research 4 Chapter 2. Generating daily gap-filled BRDF adjusted surface reflectance product at 10 m resolution using geostationary satellite product for monitoring daily canopy photosynthesis 6 1. Introduction 6 2. Methods 11 2.1 Study sites 11 2.2 In situ measurements 13 2.3 Satellite products 14 2.4 Integrated system 17 2.5 Canopy photosynthesis 21 2.6 Evaluation 23 3. Results and discussion 24 3.1 Comparison of STIF NDVI and NIRv with in situ NDVI and NIRv 24 3.2 Comparison of STIF NIRvP with in situ NIRvP 28 4. Conclusion 31 Chapter 3. Super-resolution of historic Landsat imagery using a dual Generative Adversarial Network (GAN) model with CubeSat constellation imagery for monitoring vegetation changes 32 1. Introduction 32 2. Methods 38 2.1 Real-ESRGAN model 38 2.2 Study sites 40 2.3 In situ measurements 42 2.4 Vegetation index 44 2.5 Satellite data 45 2.6 Planet Fusion 48 2.7 Dual RSS-GAN via fine-tuned Real-ESRGAN 49 2.8 Evaluation 54 3. Results 57 3.1 Comparison of NDVI and NIRv maps from Planet Fusion, Sentinel 2 NBAR, and Landsat 8 NBAR data with in situ NDVI and NIRv 57 3.2 Comparison of dual RSS-SRGAN model results with Landsat 8 NDVI and NIRv 60 3.3 Comparison of dual RSS-GAN model results with respect to in situ time-series NDVI and NIRv 63 3.4 Comparison of the dual RSS-GAN model with NDVI and NIRv maps derived from RPAS 66 4. Discussion 70 4.1 Monitoring changes in terrestrial vegetation using the dual RSS-GAN model 70 4.2 CubeSat data in the dual RSS-GAN model 72 4.3 Perspectives and limitations 73 5. Conclusion 78 Appendices 79 Supplementary material 82 Chapter 4. Matching high resolution satellite data and flux tower footprints improves their agreement in photosynthesis estimates 85 1. Introduction 85 2. Methods 89 2.1 Study sites 89 2.2 In situ measurements 92 2.3 Planet Fusion NIRvP 94 2.4 Flux footprint model 98 2.5 Evaluation 98 3. Results 105 3.1 Comparison of Planet Fusion NIRv and NIRvP with in situ NIRv and NIRvP 105 3.2 Comparison of instantaneous Planet Fusion NIRv and NIRvP with against tower GPP estimates 108 3.3 Daily GPP estimation from Planet Fusion -derived NIRvP 114 4. Discussion 118 4.1 Flux tower footprint matching and effects of spatial and temporal resolution on GPP estimation 118 4.2 Roles of radiation component in GPP mapping 123 4.3 Limitations and perspectives 126 5. Conclusion 133 Appendix 135 Supplementary Materials 144 Chapter 5. Conclusion 153 Bibliography 155 Abstract in Korea 199 Acknowledgements 202박

Deep Learning based data-fusion methods for remote sensing applications

In the last years, an increasing number of remote sensing sensors have been launched to orbit around the Earth, with a continuously growing production of massive data, that are useful for a large number of monitoring applications, especially for the monitoring task. Despite modern optical sensors provide rich spectral information about Earth's surface, at very high resolution, they are weather-sensitive. On the other hand, SAR images are always available also in presence of clouds and are almost weather-insensitive, as well as daynight available, but they do not provide a rich spectral information and are severely affected by speckle "noise" that make difficult the information extraction. For the above reasons it is worth and challenging to fuse data provided by different sources and/or acquired at different times, in order to leverage on their diversity and complementarity to retrieve the target information. Motivated by the success of the employment of Deep Learning methods in many image processing tasks, in this thesis it has been faced different typical remote sensing data-fusion problems by means of suitably designed Convolutional Neural Networks