Cloud Segmentation and Classification from All-Sky Images Using Deep Learning

For transforming the energy sector towards renewable energies, solar power is regarded as one of the major resources. However, it is not uniformly available all the time, leading to fluctuations in power generation. Clouds have the highest impact on short-term temporal and spatial variability. Thus, forecasting solar irradiance strongly depends on current cloudiness conditions. As the share of solar energy in the electrical grid is increasing, so-called nowcasts (intra-minute to intra-hour forecasts) are beneficial for grid control and for reducing required storage capacities. Furthermore, the operation of concentrating solar power (CSP) plants can be optimized with high resolution spatial solar irradiance data. A common nowcast approach is to analyze ground-based sky images from All-Sky Imagers. Clouds within these images are detected and tracked to estimate current and immediate future irradiance, whereas the accuracy of these forecasts depends primarily on the quality of pixel-level cloud recognition. State-of-the-art methods are commonly restricted to binary segmentation, distinguishing between cloudy and cloudless pixels. Thereby the optical properties of different cloud types are ignored. Also, most techniques rely on threshold-based detection showing difficulties under certain atmospheric conditions. In this thesis, two deep learning approaches are presented to automatically determine cloud conditions. To identify cloudiness characteristics like a free sun disk, a multi-label classifier was implemented assigning respective labels to images. In addition, a segmentation model was developed, classifying images pixel-wise into three cloud types and cloud-free sky. For supervised training, a new dataset of 770 images was created containing ground truth labels and segmentation masks. Moreover, to take advantage of large amounts of raw data, self-supervised pretraining was applied. By defining suitable pretext tasks, representations of image data can be learned facilitating the distinction of cloud types. Two successful techniques were chosen for self-supervised learning: Inpainting- uperresolution and DeepCluster. Afterwards, the pretrained models were fine-tuned on the annotated dataset. To assess the effectiveness of self-supervision, a comparison with random initialization and pretrained ImageNet weights was conducted. Evaluation shows that segmentation in particular benefits from self-supervised learning, improving accuracy and IoU about 3% points compared to ImageNet pretraining. The best segmentation model was also evaluated on binary segmentation. Achieving an overall accuracy of 95.15%, a state-of-the art Clear-Sky-Library (CSL) is outperformed significantly by over 7% points

A survey on deep learning techniques for image and video semantic segmentation

Image semantic segmentation is more and more being of interest for computer vision and machine learning researchers. Many applications on the rise need accurate and efficient segmentation mechanisms: autonomous driving, indoor navigation, and even virtual or augmented reality systems to name a few. This demand coincides with the rise of deep learning approaches in almost every field or application target related to computer vision, including semantic segmentation or scene understanding. This paper provides a review on deep learning methods for semantic segmentation applied to various application areas. Firstly, we formulate the semantic segmentation problem and define the terminology of this field as well as interesting background concepts. Next, the main datasets and challenges are exposed to help researchers decide which are the ones that best suit their needs and goals. Then, existing methods are reviewed, highlighting their contributions and their significance in the field. We also devote a part of the paper to review common loss functions and error metrics for this problem. Finally, quantitative results are given for the described methods and the datasets in which they were evaluated, following up with a discussion of the results. At last, we point out a set of promising future works and draw our own conclusions about the state of the art of semantic segmentation using deep learning techniques.This work has been funded by the Spanish Government TIN2016-76515-R funding for the COMBAHO project, supported with Feder funds. It has also been supported by a Spanish national grant for PhD studies FPU15/04516 (Alberto Garcia-Garcia). In addition, it was also funded by the grant Ayudas para Estudios de Master e Iniciacion a la Investigacion from the University of Alicante

Deep learning in remote sensing: a review

Standing at the paradigm shift towards data-intensive science, machine learning techniques are becoming increasingly important. In particular, as a major breakthrough in the field, deep learning has proven as an extremely powerful tool in many fields. Shall we embrace deep learning as the key to all? Or, should we resist a 'black-box' solution? There are controversial opinions in the remote sensing community. In this article, we analyze the challenges of using deep learning for remote sensing data analysis, review the recent advances, and provide resources to make deep learning in remote sensing ridiculously simple to start with. More importantly, we advocate remote sensing scientists to bring their expertise into deep learning, and use it as an implicit general model to tackle unprecedented large-scale influential challenges, such as climate change and urbanization.Comment: Accepted for publication IEEE Geoscience and Remote Sensing Magazin

Learning Transferable Representations for Visual Recognition

In the last half-decade, a new renaissance of machine learning originates from the applications of convolutional neural networks to visual recognition tasks. It is believed that a combination of big curated data and novel deep learning techniques can lead to unprecedented results. However, the increasingly large training data is still a drop in the ocean compared with scenarios in the wild. In this literature, we focus on learning transferable representation in the neural networks to ensure the models stay robust, even given different data distributions. We present three exemplar topics in three chapters, respectively: zero-shot learning, domain adaptation, and generalizable adversarial attack. By zero-shot learning, we enable models to predict labels not seen in the training phase. By domain adaptation, we improve a model\u27s performance on the target domain by mitigating its discrepancy from a labeled source model, without any target annotation. Finally, the generalization adversarial attack focuses on learning an adversarial camouflage that ideally would work in every possible scenario. Despite sharing the same transfer learning philosophy, each of the proposed topics poses a unique challenge requiring a unique solution. In each chapter, we introduce the problem as well as present our solution to the problem. We also discuss some other researchers\u27 approaches and compare our solution to theirs in the experiments