31 research outputs found
Interpretable Hyperspectral AI: When Non-Convex Modeling meets Hyperspectral Remote Sensing
Hyperspectral imaging, also known as image spectrometry, is a landmark
technique in geoscience and remote sensing (RS). In the past decade, enormous
efforts have been made to process and analyze these hyperspectral (HS) products
mainly by means of seasoned experts. However, with the ever-growing volume of
data, the bulk of costs in manpower and material resources poses new challenges
on reducing the burden of manual labor and improving efficiency. For this
reason, it is, therefore, urgent to develop more intelligent and automatic
approaches for various HS RS applications. Machine learning (ML) tools with
convex optimization have successfully undertaken the tasks of numerous
artificial intelligence (AI)-related applications. However, their ability in
handling complex practical problems remains limited, particularly for HS data,
due to the effects of various spectral variabilities in the process of HS
imaging and the complexity and redundancy of higher dimensional HS signals.
Compared to the convex models, non-convex modeling, which is capable of
characterizing more complex real scenes and providing the model
interpretability technically and theoretically, has been proven to be a
feasible solution to reduce the gap between challenging HS vision tasks and
currently advanced intelligent data processing models
Spatial-Spectral Manifold Embedding of Hyperspectral Data
In recent years, hyperspectral imaging, also known as imaging spectroscopy,
has been paid an increasing interest in geoscience and remote sensing
community. Hyperspectral imagery is characterized by very rich spectral
information, which enables us to recognize the materials of interest lying on
the surface of the Earth more easier. We have to admit, however, that high
spectral dimension inevitably brings some drawbacks, such as expensive data
storage and transmission, information redundancy, etc. Therefore, to reduce the
spectral dimensionality effectively and learn more discriminative spectral
low-dimensional embedding, in this paper we propose a novel hyperspectral
embedding approach by simultaneously considering spatial and spectral
information, called spatial-spectral manifold embedding (SSME). Beyond the
pixel-wise spectral embedding approaches, SSME models the spatial and spectral
information jointly in a patch-based fashion. SSME not only learns the spectral
embedding by using the adjacency matrix obtained by similarity measurement
between spectral signatures, but also models the spatial neighbours of a target
pixel in hyperspectral scene by sharing the same weights (or edges) in the
process of learning embedding. Classification is explored as a potential
strategy to quantitatively evaluate the performance of learned embedding
representations. Classification is explored as a potential application for
quantitatively evaluating the performance of these hyperspectral embedding
algorithms. Extensive experiments conducted on the widely-used hyperspectral
datasets demonstrate the superiority and effectiveness of the proposed SSME as
compared to several state-of-the-art embedding methods
More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification
Classification and identification of the materials lying over or beneath the
Earth's surface have long been a fundamental but challenging research topic in
geoscience and remote sensing (RS) and have garnered a growing concern owing to
the recent advancements of deep learning techniques. Although deep networks
have been successfully applied in single-modality-dominated classification
tasks, yet their performance inevitably meets the bottleneck in complex scenes
that need to be finely classified, due to the limitation of information
diversity. In this work, we provide a baseline solution to the aforementioned
difficulty by developing a general multimodal deep learning (MDL) framework. In
particular, we also investigate a special case of multi-modality learning (MML)
-- cross-modality learning (CML) that exists widely in RS image classification
applications. By focusing on "what", "where", and "how" to fuse, we show
different fusion strategies as well as how to train deep networks and build the
network architecture. Specifically, five fusion architectures are introduced
and developed, further being unified in our MDL framework. More significantly,
our framework is not only limited to pixel-wise classification tasks but also
applicable to spatial information modeling with convolutional neural networks
(CNNs). To validate the effectiveness and superiority of the MDL framework,
extensive experiments related to the settings of MML and CML are conducted on
two different multimodal RS datasets. Furthermore, the codes and datasets will
be available at https://github.com/danfenghong/IEEE_TGRS_MDL-RS, contributing
to the RS community
Changes from Classical Statistics to Modern Statistics and Data Science
A coordinate system is a foundation for every quantitative science,
engineering, and medicine. Classical physics and statistics are based on the
Cartesian coordinate system. The classical probability and hypothesis testing
theory can only be applied to Euclidean data. However, modern data in the real
world are from natural language processing, mathematical formulas, social
networks, transportation and sensor networks, computer visions, automations,
and biomedical measurements. The Euclidean assumption is not appropriate for
non Euclidean data. This perspective addresses the urgent need to overcome
those fundamental limitations and encourages extensions of classical
probability theory and hypothesis testing , diffusion models and stochastic
differential equations from Euclidean space to non Euclidean space. Artificial
intelligence such as natural language processing, computer vision, graphical
neural networks, manifold regression and inference theory, manifold learning,
graph neural networks, compositional diffusion models for automatically
compositional generations of concepts and demystifying machine learning
systems, has been rapidly developed. Differential manifold theory is the
mathematic foundations of deep learning and data science as well. We urgently
need to shift the paradigm for data analysis from the classical Euclidean data
analysis to both Euclidean and non Euclidean data analysis and develop more and
more innovative methods for describing, estimating and inferring non Euclidean
geometries of modern real datasets. A general framework for integrated analysis
of both Euclidean and non Euclidean data, composite AI, decision intelligence
and edge AI provide powerful innovative ideas and strategies for fundamentally
advancing AI. We are expected to marry statistics with AI, develop a unified
theory of modern statistics and drive next generation of AI and data science.Comment: 37 page
Spatial-Spectral Manifold Embedding of Hyperspectral Data
In recent years, hyperspectral imaging, also known as imaging spectroscopy, has been paid an increasing interest in geoscience and
remote sensing community. Hyperspectral imagery is characterized by very rich spectral information, which enables us to recognize the materials of interest lying on the surface of the Earth more easier. We have to admit, however, that high spectral dimension inevitably brings some drawbacks, such as expensive data storage and transmission, information redundancy, etc. Therefore, to reduce the spectral dimensionality effectively and learn more discriminative spectral low-dimensional embedding, in this paper we propose a novel hyperspectral embedding approach by simultaneously considering spatial and spectral information, called spatialspectral manifold embedding (SSME). Beyond the pixel-wise spectral embedding approaches, SSME models the spatial and spectral information jointly in a patch-based fashion. SSME not only learns the spectral embedding by using the adjacency matrix obtained by similarity measurement between spectral signatures, but also models the spatial neighbours of a target pixel in hyperspectral scene by sharing the same weights (or edges) in the process of learning embedding. Classification is explored as a potential strategy
to quantitatively evaluate the performance of learned embedding representations. Classification is explored as a potential application for quantitatively evaluating the performance of these hyperspectral embedding algorithms. Extensive experiments conducted on the widely-used hyperspectral datasets demonstrate the superiority and effectiveness of the proposed SSME as compared to several state-of-the-art embedding methods
X-ModalNet: A Semi-Supervised Deep Cross-Modal Network for Classification of Remote Sensing Data
This paper addresses the problem of semi-supervised transfer learning with
limited cross-modality data in remote sensing. A large amount of multi-modal
earth observation images, such as multispectral imagery (MSI) or synthetic
aperture radar (SAR) data, are openly available on a global scale, enabling
parsing global urban scenes through remote sensing imagery. However, their
ability in identifying materials (pixel-wise classification) remains limited,
due to the noisy collection environment and poor discriminative information as
well as limited number of well-annotated training images. To this end, we
propose a novel cross-modal deep-learning framework, called X-ModalNet, with
three well-designed modules: self-adversarial module, interactive learning
module, and label propagation module, by learning to transfer more
discriminative information from a small-scale hyperspectral image (HSI) into
the classification task using a large-scale MSI or SAR data. Significantly,
X-ModalNet generalizes well, owing to propagating labels on an updatable graph
constructed by high-level features on the top of the network, yielding
semi-supervised cross-modality learning. We evaluate X-ModalNet on two
multi-modal remote sensing datasets (HSI-MSI and HSI-SAR) and achieve a
significant improvement in comparison with several state-of-the-art methods
Learning to Propagate Labels on Graphs: An Iterative Multitask Regression Framework for Semi-supervised Hyperspectral Dimensionality Reduction
Hyperspectral dimensionality reduction (HDR), an important preprocessing step prior to high-level data analysis, has been garnering growing attention in the remote sensing community. Although a variety of methods, both unsupervised and supervised models, have been proposed for this task, yet the discriminative ability in feature representation still remains limited due to the lack of a powerful tool that effectively exploits the labeled and unlabeled data in the HDR process. A semi-supervised HDR approach, called iterative multitask regression (IMR), is proposed in this paper to address this need. IMR aims at learning a low-dimensional subspace by jointly considering the labeled and unlabeled data, and also bridging the learned subspace with two regression tasks: labels and pseudo-labels initialized by a given classifier. More significantly, IMR dynamically propagates the labels on a learnable graph and progressively refines pseudo-labels, yielding a well-conditioned feedback system. Experiments conducted on three widely-used hyperspectral image datasets demonstrate that the dimension-reduced features learned by the proposed IMR framework with respect to classification or recognition accuracy are superior to those of related state-of-the-art HDR approaches
Coupled Convolutional Neural Network with Adaptive Response Function Learning for Unsupervised Hyperspectral Super-Resolution
Due to the limitations of hyperspectral imaging systems, hyperspectral
imagery (HSI) often suffers from poor spatial resolution, thus hampering many
applications of the imagery. Hyperspectral super-resolution refers to fusing
HSI and MSI to generate an image with both high spatial and high spectral
resolutions. Recently, several new methods have been proposed to solve this
fusion problem, and most of these methods assume that the prior information of
the Point Spread Function (PSF) and Spectral Response Function (SRF) are known.
However, in practice, this information is often limited or unavailable. In this
work, an unsupervised deep learning-based fusion method - HyCoNet - that can
solve the problems in HSI-MSI fusion without the prior PSF and SRF information
is proposed. HyCoNet consists of three coupled autoencoder nets in which the
HSI and MSI are unmixed into endmembers and abundances based on the linear
unmixing model. Two special convolutional layers are designed to act as a
bridge that coordinates with the three autoencoder nets, and the PSF and SRF
parameters are learned adaptively in the two convolution layers during the
training process. Furthermore, driven by the joint loss function, the proposed
method is straightforward and easily implemented in an end-to-end training
manner. The experiments performed in the study demonstrate that the proposed
method performs well and produces robust results for different datasets and
arbitrary PSFs and SRFs
A Multimodal Feature Selection Method for Remote Sensing Data Analysis Based on Double Graph Laplacian Diagonalization
When dealing with multivariate remotely sensed records collected by multiple sensors, an accurate selection of information at the data, feature, or decision level is instrumental in improving the scenes’ characterization. This will also enhance the system’s efficiency and provide more details on modeling the physical phenomena occurring on the Earth’s surface. In this article, we introduce a flexible and efficient method based on graph Laplacians for information selection at different levels of data fusion. The proposed approach combines data structure and information content to address the limitations of existing graph-Laplacian-based methods in dealing with heterogeneous datasets. Moreover, it adapts the selection to each homogenous area of the considered images according to their underlying properties. Experimental tests carried out on several multivariate remote sensing datasets show the consistency of the proposed approach