Sparse + smooth decomposition models for multi-temporal SAR images

International audienceSAR images have distinctive characteristics compared to optical images: speckle phenomenon produces strong fluctuations, and strong scatterers have radar signatures several orders of magnitude larger than others. We propose to use an image decomposition approach to account for these peculiarities. Several methods have been proposed in the field of image processing to decompose an image into components of different nature, such as a geometrical part and a textural part. They are generally stated as an energy minimization problem where specific penalty terms are applied to each component of the sought decomposition. We decompose temporal series of SAR images into three components: speckle, strong scatterers and background. Our decomposition method is based on a discrete optimization technique by graph-cut. We apply it to change detection tasks

RSVQA: Visual Question Answering for Remote Sensing Data

This paper introduces the task of visual question answering for remote sensing data (RSVQA). Remote sensing images contain a wealth of information which can be useful for a wide range of tasks including land cover classification, object counting or detection. However, most of the available methodologies are task-specific, thus inhibiting generic and easy access to the information contained in remote sensing data. As a consequence, accurate remote sensing product generation still requires expert knowledge. With RSVQA, we propose a system to extract information from remote sensing data that is accessible to every user: we use questions formulated in natural language and use them to interact with the images. With the system, images can be queried to obtain high level information specific to the image content or relational dependencies between objects visible in the images. Using an automatic method introduced in this article, we built two datasets (using low and high resolution data) of image/question/answer triplets. The information required to build the questions and answers is queried from OpenStreetMap (OSM). The datasets can be used to train (when using supervised methods) and evaluate models to solve the RSVQA task. We report the results obtained by applying a model based on Convolutional Neural Networks (CNNs) for the visual part and on a Recurrent Neural Network (RNN) for the natural language part to this task. The model is trained on the two datasets, yielding promising results in both cases.Comment: 12 pages, Published in IEEE Transactions on Geoscience and Remote Sensing. Added one experiment and authors' biographie

Deep learning for classification of noisy QR codes

We wish to define the limits of a classical classification model based on deep learning when applied to abstract images, which do not represent visually identifiable objects.QR codes (Quick Response codes) fall into this category of abstract images: one bit corresponding to one encoded character, QR codes were not designed to be decoded manually. To understand the limitations of a deep learning-based model for abstract image classification, we train an image classification model on QR codes generated from information obtained when reading a health pass. We compare a classification model with a classical (deterministic) decoding method in the presence of noise. This study allows us to conclude that a model based on deep learning can be relevant for the understanding of abstract images.Comment: in French language. RFIAP 2022 - Reconnaissance des Formes, Image, Apprentissage et Perception, Jul 2022, Vannes (Bretagne), Franc

The curse of language biases in remote sensing VQA: the role of spatial attributes, language diversity, and the need for clear evaluation

Remote sensing visual question answering (RSVQA) opens new opportunities for the use of overhead imagery by the general public, by enabling human-machine interaction with natural language. Building on the recent advances in natural language processing and computer vision, the goal of RSVQA is to answer a question formulated in natural language about a remote sensing image. Language understanding is essential to the success of the task, but has not yet been thoroughly examined in RSVQA. In particular, the problem of language biases is often overlooked in the remote sensing community, which can impact model robustness and lead to wrong conclusions about the performances of the model. Thus, the present work aims at highlighting the problem of language biases in RSVQA with a threefold analysis strategy: visual blind models, adversarial testing and dataset analysis. This analysis focuses both on model and data. Moreover, we motivate the use of more informative and complementary evaluation metrics sensitive to the issue. The gravity of language biases in RSVQA is then exposed for all of these methods with the training of models discarding the image data and the manipulation of the visual input during inference. Finally, a detailed analysis of question-answer distribution demonstrates the root of the problem in the data itself. Thanks to this analytical study, we observed that biases in remote sensing are more severe than in standard VQA, likely due to the specifics of existing remote sensing datasets for the task, e.g. geographical similarities and sparsity, as well as a simpler vocabulary and question generation strategies. While new, improved and less-biased datasets appear as a necessity for the development of the promising field of RSVQA, we demonstrate that more informed, relative evaluation metrics remain much needed to transparently communicate results of future RSVQA methods

Learning multi-label aerial image classification under label noise: a regularization approach using word embeddings

Training deep neural networks requires well-annotated datasets. However, real world datasets are often noisy, especially in a multi-label scenario, i.e. where each data point can be attributed to more than one class. To this end, we propose a regularization method to learn multi-label classification networks from noisy data. This regularization is based on the assumption that semantically close classes are more likely to appear together in a given image. Hereby, we encode label correlations with prior knowledge and regularize noisy network predictions using label correlations. To evaluate its effectiveness, we perform experiments on a mutli-label aerial image dataset contaminated with controlled levels of label noise. Results indicate that networks trained using the proposed method outperform those directly learned from noisy labels and that the benefits increase proportionally to the amount of noise present

Markovian models for SAR images : application to water detection in SWOT satellite images and multi-temporal analysis of urban areas

Afin d’obtenir une meilleure couverture, à la fois spatiale et temporelle de leurs mesures les hydrologues utilisent des données spatiales en plus de celles acquises sur place. Fruit d’une collaboration entre les agences spatiales française (le CNES) et américaine (JPL, NASA), la future mission SWOT a notamment pour but de fournir des mesures de hauteur des surfaces d’eau continentales en utilisant l’interférométrie radar à synthèse d’ouverture (SAR). Dans cette thèse, nous nous intéressons au problème de la détection de l’eau dans les images d’amplitude SWOT qui est ici un prérequis au traitement interférométrique. Dans cette optique, nous proposons d’utiliser une méthode dédiée à la détection des larges cours d’eau ainsi qu’un traitement spécifique pour la détection de rivières fines. La première méthode est basée sur un champ de Markov (MRF) pour la classification, conjointement à une estimation des paramètres de classes qui ne peuvent être supposés constants dans le cas de SWOT. L’estimation des paramètres peut également être modélisée par des champs de Markov. La seconde méthode s’appuie sur une détection de segments au niveau pixellique complétée par une connexion de ces segments. Afin d’étudier l’extension aux séries multi-temporelles, nous proposons des méthodes de traitement adaptées aux données SAR de zones urbaines. Ces zones présentent de forts rétro-diffuseurs, ayant une radiométrie largement supérieure à celle des autres points dans l’image. Les modèles présentés prennent explicitement en compte la présence de ces forts rétro-diffuseurs en considérant les images comme une somme de deux composantes (le fond et les cibles fortes). Différents termes de régularisation peuvent alors être utilisés pour chacune de ces deux composantes. Modélisés comme des champs de Markov, ils peuvent alors être optimisés exactement par recherche de coupure minimale dans un graphe. Nous présentons des applications en détection de cibles fortes, régularisation et détection de changement dans ces séries.To obtain a better coverage both spatially and temporally, hydrologists use spaceborne data in addition to data acquired in situ. Resulting from a collaboration between NASA’s Jet Propulsion Laboratory (JPL) and the French Space Agency (CNES), the upcoming SWOT mission will provide global continental water elevation measures using Synthetic Aperture Radar (SAR) interferometry. In this dissertation, we address the problem of water detection in SWOT amplitude images, which is to be performed before the interferometric processing. To this end, we propose to use a method dedicated to the detection of large water bodies and a specific algorithm for the detection of narrow rivers. The first method is based on Markov Random Fields (MRF). The classification is regularized and the class parameters, which cannot be assumed constant in the case of SWOT, are jointly estimated. The second method is based on segment detection at the pixel level, completed by a connection step. To study the extension to multi-temporal data, we propose methods adapted to the processing of series of SAR images of urban areas. These areas feature strong scatterers, having a radiometry orders of magnitude higher than the other points in the image. The proposed models explicitly account for the presence of these strong scatterers by considering the images as a sum of two components (the background and the strong scatterers). Different regularization terms can then be applied to each of these components. Modeled as MRF, they can then be optimized exactly using graph cuts. We present applications for strong scatterers detection, regularization and change detection

Multitemporal SAR Image Decomposition into Strong Scatterers, Background, and Speckle

International audienceSpeckle phenomenon in synthetic aperture radar (SAR) images makes their visual and automatic interpretation a difficult task. To reduce strong fluctuations due to speckle, total variation (TV) regularization has been proposed by several authors to smooth out noise without blurring edges. A specificity of SAR images is the presence of strong scatterers having a radiometry several orders of magnitude larger than their surrounding region. These scatterers, especially present in urban areas, limit the effectiveness of TV regularization as they break the assumption of an image made of regions of constant radiometry. To overcome this limitation, we propose in this paper an image decomposition approach. There exists numerous methods to decompose an image into several components, notably to separate textural and geometrical information. These decomposition models are generally recast as energy minimization problems involving a different penalty term for each of the components. In this framework, we propose an energy suitable for the decomposition of SAR images into speckle, a smooth background and strong scatterers, and discuss its minimization using max-flow/min-cut algorithms. We make the connection between the minimization problem considered, involving the L0 pseudo-norm, and the generalized likelihood ratio test used in detection theory. The proposed decomposition jointly performs the detection of strong scatterers and the estimation of the background radiometry. Given the increasing availability of time series of SAR images, we consider the decomposition of a whole time series. New change detection methods can be based on the temporal analysis of the components obtained from our decomposition

Modèles Markoviens pour les images SAR : application à la détection de l'eau dans les images satellitaires SWOT et analyse multi-temporelle de zones urbaines

To obtain a better coverage both spatially and temporally, hydrologists use spaceborne data in addition to data acquired in situ. Resulting from a collaboration between NASA’s Jet Propulsion Laboratory (JPL) and the French Space Agency (CNES), the upcoming SWOT mission will provide global continental water elevation measures using Synthetic Aperture Radar (SAR) interferometry. In this dissertation, we address the problem of water detection in SWOT amplitude images, which is to be performed before the interferometric processing. To this end, we propose to use a method dedicated to the detection of large water bodies and a specific algorithm for the detection of narrow rivers. The first method is based on Markov Random Fields (MRF). The classification is regularized and the class parameters, which cannot be assumed constant in the case of SWOT, are jointly estimated. The second method is based on segment detection at the pixel level, completed by a connection step. To study the extension to multi-temporal data, we propose methods adapted to the processing of series of SAR images of urban areas. These areas feature strong scatterers, having a radiometry orders of magnitude higher than the other points in the image. The proposed models explicitly account for the presence of these strong scatterers by considering the images as a sum of two components (the background and the strong scatterers). Different regularization terms can then be applied to each of these components. Modeled as MRF, they can then be optimized exactly using graph cuts. We present applications for strong scatterers detection, regularization and change detection.Afin d’obtenir une meilleure couverture, à la fois spatiale et temporelle de leurs mesures les hydrologues utilisent des données spatiales en plus de celles acquises sur place. Fruit d’une collaboration entre les agences spatiales française (le CNES) et américaine (JPL, NASA), la future mission SWOT a notamment pour but de fournir des mesures de hauteur des surfaces d’eau continentales en utilisant l’interférométrie radar à synthèse d’ouverture (SAR). Dans cette thèse, nous nous intéressons au problème de la détection de l’eau dans les images d’amplitude SWOT qui est ici un prérequis au traitement interférométrique. Dans cette optique, nous proposons d’utiliser une méthode dédiée à la détection des larges cours d’eau ainsi qu’un traitement spécifique pour la détection de rivières fines. La première méthode est basée sur un champ de Markov (MRF) pour la classification, conjointement à une estimation des paramètres de classes qui ne peuvent être supposés constants dans le cas de SWOT. L’estimation des paramètres peut également être modélisée par des champs de Markov. La seconde méthode s’appuie sur une détection de segments au niveau pixellique complétée par une connexion de ces segments. Afin d’étudier l’extension aux séries multi-temporelles, nous proposons des méthodes de traitement adaptées aux données SAR de zones urbaines. Ces zones présentent de forts rétro-diffuseurs, ayant une radiométrie largement supérieure à celle des autres points dans l’image. Les modèles présentés prennent explicitement en compte la présence de ces forts rétro-diffuseurs en considérant les images comme une somme de deux composantes (le fond et les cibles fortes). Différents termes de régularisation peuvent alors être utilisés pour chacune de ces deux composantes. Modélisés comme des champs de Markov, ils peuvent alors être optimisés exactement par recherche de coupure minimale dans un graphe. Nous présentons des applications en détection de cibles fortes, régularisation et détection de changement dans ces séries