Deep internal learning for inpainting of cloud-affected regions in satellite imagery

Cloud cover remains a significant limitation to a broad range of applications relying on optical remote sensing imagery, including crop identification/yield prediction, climate monitoring, and land cover classification. A common approach to cloud removal treats the problem as an inpainting task and imputes optical data in the cloud-affected regions employing either mosaicing historical data or making use of sensing modalities not impacted by cloud obstructions, such as SAR. Recently, deep learning approaches have been explored in these applications; however, the majority of reported solutions rely on external learning practices, i.e., models trained on fixed datasets. Although these models perform well within the context of a particular dataset, a significant risk of spatial and temporal overfitting exists when applied in different locations or at different times. Here, cloud removal was implemented within an internal learning regime through an inpainting technique based on the deep image prior. The approach was evaluated on both a synthetic dataset with an exact ground truth, as well as real samples. The ability to inpaint the cloud-affected regions for varying weather conditions across a whole year with no prior training was demonstrated, and the performance of the approach was characterised

Weakly Supervised Learning for Multi-Image Synthesis

Machine learning-based approaches have been achieving state-of-the-art results on many computer vision tasks. While deep learning and convolutional networks have been incredibly popular, these approaches come at the expense of huge amounts of labeled data required for training. Manually annotating large amounts of data, often millions of images in a single dataset, is costly and time consuming. To deal with the problem of data annotation, the research community has been exploring approaches that require less amount of labelled data. The central problem that we consider in this research is image synthesis without any manual labeling. Image synthesis is a classic computer vision task that requires understanding of image contents and their semantic and geometric properties. We propose that we can train image synthesis models by relying on sequences of videos and using weakly supervised learning. Large amounts of unlabeled data are freely available on the internet. We propose to set up the training in a multi-image setting so that we can use one of the images as the target - this allows us to rely only on images for training and removes the need for manual annotations. We demonstrate three main contributions in this work. First, we present a method of fusing multiple noisy overhead images to make a single, artifact-free image. We present a weakly supervised method that relies on crowd-sourced labels from online maps and a completely unsupervised variant that only requires a series of satellite images as inputs. Second, we propose a single-image novel view synthesis method for complex, outdoor scenes. We propose a learning-based method that uses pairs of nearby images captured on urban roads and their respective GPS coordinates as supervision. We show that a model trained with this automatically captured data can render a new view of a scene that can be as far as 10 meters from the input image. Third, we consider the problem of synthesizing new images of a scene under different conditions, such as time of day and season, based on a single input image. As opposed to existing methods, we do not need manual annotations for transient attributes, such as fog or snow, for training. We train our model by using streams of images captured from outdoor webcams and time-lapse videos. Through these applications, we show several settings where we can train state-of-the-art deep learning methods without manual annotations. This work focuses on three image synthesis tasks. We propose weakly supervised learning and remove requirements for manual annotations by relying on sequences of images. Our approach is in line with the research efforts that aim to minimize the labels required for training machine learning methods

Cloud and Cloud Shadow Segmentation for Remote Sensing Imagery via Filtered Jaccard Loss Function and Parametric Augmentation

Cloud and cloud shadow segmentation are fundamental processes in optical remote sensing image analysis. Current methods for cloud/shadow identification in geospatial imagery are not as accurate as they should, especially in the presence of snow and haze. This paper presents a deep learning-based framework for the detection of cloud/shadow in Landsat 8 images. Our method benefits from a convolutional neural network, Cloud-Net+ (a modification of our previously proposed Cloud-Net) that is trained with a novel loss function (Filtered Jaccard Loss). The proposed loss function is more sensitive to the absence of foreground objects in an image and penalizes/rewards the predicted mask more accurately than other common loss functions. In addition, a sunlight direction-aware data augmentation technique is developed for the task of cloud shadow detection to extend the generalization ability of the proposed model by expanding existing training sets. The combination of Cloud-Net+, Filtered Jaccard Loss function, and the proposed augmentation algorithm delivers superior results on four public cloud/shadow detection datasets. Our experiments on Pascal VOC dataset exemplifies the applicability and quality of our proposed network and loss function in other computer vision applications

Explaining the Effects of Clouds on Remote Sensing Scene Classification

Most of Earth is covered by haze or clouds, impeding the constant monitoring of our planet. Preceding works have documented the detrimental effects of cloud coverage on remote sensing applications and proposed ways to approach this issue. However, up to now, little effort has been spent on understanding how exactly atmospheric disturbances impede the application of modern machine learning methods to Earth observation data. Specifically, we consider the effects of haze and cloud coverage on a scene classification task. We provide a thorough investigation of how classifiers trained on cloud-free data fail once they encounter noisy imagery—a common scenario encountered when deploying pretrained models for remote sensing to real use cases. We show how and why remote sensing scene classification suffers from cloud coverage. Based on a multistage analysis, including explainability approaches applied to the predictions, we work out four different types of effects that clouds have on scene prediction. The contribution of our work is to deepen the understanding of the effects of clouds on common remote sensing applications and consequently guide the development of more robust methods

Land Cover mapping based on Hierarchical Decision Trees

The ability to monitor land cover changes can be very useful for resource management, urban planning, forest fire identification, among plenty of other applications. The topic of remote sensing has been studied for a long time, with many different solutions that typically use satellites or aircraft to obtain multi-spectral imagery and further analyse it. The entity responsible for monitoring land use and land cover in Portugal is Direção- Geral do Território (DGT) which periodically produces a document called Land Use and Land Cover Map (Carta de Uso e Ocupação do Solo (COS), in Portuguese). This document uses imagery with high spatial resolution of 0,25 m and has a minimum mapping unit of 1 ha, however, it is only produced every few years because it is manually curated by experts. This hinders the ability to closely monitor relevant land changes that occur more frequently or rapidly. In this dissertation, several classifiers were developed in a hierarchical manner to address some of COS drawbacks. The classifiers used were based on decision trees which were trained using satellite imagery collected from Sentinel-2 satellite constellation. Although having a lower spatial resolution than COS, they can automatically classify land cover in some minutes every time a new set of Sentinel-2 imagery is collected, in this case each 5 days. Cloud coverage might make some of these images unusable but nonetheless, the temporal resolution is still far greater than COS. However, automatic classification is not as accurate as manual classification. The produced classifiers did not consider as many classes as COS and had problems distinguishing some types of land cover, due to either poor sample size or spectral signature similarity. Considering Matthews Correlation Coefficient (MCC), water class had the best performance with an average of 91,28%, followed by forest and agriculture class with an average of 47,88% and 42,34%, respectively, and lastly urban areas and bare land class had the worse results averaging 28,03% and 20,53% respectively. Nevertheless, the results obtained were still considered to be good, but with considerable room for improvement.Acompanhar as mudanças de ocupação de solo tem bastante utilidade para uma correta gestão de recursos, deteção de fogos florestais, e inúmeras aplicações. O tema de deteção remota é estudado há vários anos e tipicamente são usadas imagens multiespectrais obtidas através de satélites e aeronaves que são depois analisadas em detalhe. A entidade responsável por esta monitorização em Portugal é a Direção-Geral do Território (DGT) que produz a Carta de Uso e Ocupação do Solo (COS), onde identifica o uso e ocupação de solo de Portugal continental. Este documento tem uma resolução espacial muito boa mas a sua resolução temporal é muito baixa, pois só é produzido em alguns anos visto ser feito de forma manual. Isto é prejudicial ao acompanhamento em detalhe das mudanças na ocupação de solo visto muita informação não ser registada. Nesta dissertação desenvolveram-se vários classificadores, distribuídos de forma hierárquica, para mitigar este problema. Foram usadas árvores de decisão treinadas com imagens recolhidas pela constelação Sentinel-2. Apesar destas imagens terem uma resolução espacial mais fraca, os classificadores conseguem classificar o solo de maneira automática apenas em alguns minutos cada vez que um novo conjunto de imagens é recolhido, neste caso a cada 5 dias. Nem todas as imagens podem ser usadas, devido às condições atmosféricas, mas continua a ter uma resolução temporal superior à COS. No entanto, esta classificação automática não é tão exata quanto a manual. Também não foram consideradas tantas classes quanto as presentes na COS e os classificadores tiveram dificuldade em diferenciar algumas delas, seja pela amostra ser muito pequena ou pelos valores espetrais serem demasiado semelhantes. Considerando o Matthews Correlation Coefficient (MCC), a classe “water” obteve os melhores resultados com uma média de 91,28%, seguida pelas classes “forest” e “agriculture” com uma média de 47,88% e 42,34%, respetivamente, e por último as classes “urban areas” e “bare land” com uma média de 28,03% e 20,53% respetivamente. Mesmo assim considera-se que os resultados obtidos são satisfatórios, mas com muitas oportunidades de melhoria

Semantic location extraction from crowdsourced data

Crowdsourced Data (CSD) has recently received increased attention in many application areas including disaster management. Convenience of production and use, data currency and abundancy are some of the key reasons for attracting this high interest. Conversely, quality issues like incompleteness, credibility and relevancy prevent the direct use of such data in important applications like disaster management. Moreover, location information availability of CSD is problematic as it remains very low in many crowd sourced platforms such as Twitter. Also, this recorded location is mostly related to the mobile device or user location and often does not represent the event location. In CSD, event location is discussed descriptively in the comments in addition to the recorded location (which is generated by means of mobile device's GPS or mobile communication network). This study attempts to semantically extract the CSD location information with the help of an ontological Gazetteer and other available resources. 2011 Queensland flood tweets and Ushahidi Crowd Map data were semantically analysed to extract the location information with the support of Queensland Gazetteer which is converted to an ontological gazetteer and a global gazetteer. Some preliminary results show that the use of ontologies and semantics can improve the accuracy of place name identification of CSD and the process of location information extraction

Remote Sensing of the Aquatic Environments

The book highlights recent research efforts in the monitoring of aquatic districts with remote sensing observations and proximal sensing technology integrated with laboratory measurements. Optical satellite imagery gathered at spatial resolutions down to few meters has been used for quantitative estimations of harmful algal bloom extent and Chl-a mapping, as well as winds and currents from SAR acquisitions. The knowledge and understanding gained from this book can be used for the sustainable management of bodies of water across our planet

Calibration of DART Radiative Transfer Model with Satellite Images for Simulating Albedo and Thermal Irradiance Images and 3D Radiative Budget of Urban Environment

Remote sensing is increasingly used for managing urban environment. In this context, the H2020 project URBANFLUXES aims to improve our knowledge on urban anthropogenic heat fluxes, with the specific study of three cities: London, Basel and Heraklion. Usually, one expects to derive directly 2 major urban parameters from remote sensing: the albedo and thermal irradiance. However, the determination of these two parameters is seriously hampered by complexity of urban architecture. For example, urban reflectance and brightness temperature are far from isotropic and are spatially heterogeneous. Hence, radiative transfer models that consider the complexity of urban architecture when simulating remote sensing signals are essential tools. Even for these sophisticated models, there is a major constraint for an operational use of remote sensing: the complex 3D distribution of optical properties and temperatures in urban environments. Here, the work is conducted with the DART (Discrete Anisotropic Radiative Transfer) model. It is a comprehensive physically based 3D radiative transfer model that simulates optical signals at the entrance of imaging spectro-radiometers and LiDAR scanners on board of satellites and airplanes, as well as the 3D radiative budget, of urban and natural landscapes for any experimental (atmosphere, topography,…) and instrumental (sensor altitude, spatial resolution, UV to thermal infrared,…) configuration. Paul Sabatier University distributes free licenses for research activities. This paper presents the calibration of DART model with high spatial resolution satellite images (Landsat 8, Sentinel 2, etc.) that are acquired in the visible (VIS) / near infrared (NIR) domain and in the thermal infrared (TIR) domain. Here, the work is conducted with an atmospherically corrected Landsat 8 image and Bale city, with its urban database. The calibration approach in the VIS/IR domain encompasses 5 steps for computing the 2D distribution (image) of urban albedo at satellite spatial resolution. (1) DART simulation of satellite image at very high spatial resolution (e.g., 50cm) per satellite spectral band. Atmosphere conditions are specific to the satellite image acquisition. (2) Spatial resampling of DART image at the coarser spatial resolution of the available satellite image, per spectral band. (3) Iterative derivation of the urban surfaces (roofs, walls, streets, vegetation,…) optical properties as derived from pixel-wise comparison of DART and satellite images, independently per spectral band. (4) Computation of the band albedo image of the city, per spectral band. (5) Computation of the image of the city albedo and VIS/NIR exitance, as an integral over all satellite spectral bands. In order to get a time series of albedo and VIS/NIR exitance, even in the absence of satellite images, ECMWF information about local irradiance and atmosphere conditions are used. A similar approach is used for calculating the city thermal exitance using satellite images acquired in the thermal infrared domain. Finally, DART simulations that are conducted with the optical properties derived from remote sensing images give also the 3D radiative budget of the city at any date including the date of the satellite image acquisition