90 research outputs found
Cross-Attention in Coupled Unmixing Nets for Unsupervised Hyperspectral Super-Resolution
The recent advancement of deep learning techniques has made great progress on
hyperspectral image super-resolution (HSI-SR). Yet the development of
unsupervised deep networks remains challenging for this task. To this end, we
propose a novel coupled unmixing network with a cross-attention mechanism,
CUCaNet for short, to enhance the spatial resolution of HSI by means of
higher-spatial-resolution multispectral image (MSI). Inspired by coupled
spectral unmixing, a two-stream convolutional autoencoder framework is taken as
backbone to jointly decompose MS and HS data into a spectrally meaningful basis
and corresponding coefficients. CUCaNet is capable of adaptively learning
spectral and spatial response functions from HS-MS correspondences by enforcing
reasonable consistency assumptions on the networks. Moreover, a cross-attention
module is devised to yield more effective spatial-spectral information transfer
in networks. Extensive experiments are conducted on three widely-used HS-MS
datasets in comparison with state-of-the-art HSI-SR models, demonstrating the
superiority of the CUCaNet in the HSI-SR application. Furthermore, the codes
and datasets will be available at:
https://github.com/danfenghong/ECCV2020_CUCaNet
Deep Learning for Embedding and Integrating Multimodal Biomedical Data
Biomedical data is being generated in extremely high throughput and high dimension by technologies in areas ranging from single-cell genomics, proteomics, and transcriptomics (cytometry, single-cell RNA and ATAC sequencing) to neuroscience and cognition (fMRI and PET) to pharmaceuticals (drug perturbations and interactions). These new and emerging technologies and the datasets they create give an unprecedented view into the workings of their respective biological entities. However, there is a large gap between the information contained in these datasets and the insights that current machine learning methods can extract from them. This is especially the case when multiple technologies can measure the same underlying biological entity or system. By separately analyzing the same system but from different views gathered by different data modalities, patterns are left unobserved if they only emerge from the multi-dimensional joint representation of all of the modalities together. Through an interdisciplinary approach that emphasizes active collaboration with data domain experts, my research has developed models for data integration, extracting important insights through the joint analysis of varied data sources. In this thesis, I discuss models that address this task of multi-modal data integration, especially generative adversarial networks (GANs) and autoencoders (AEs). My research has been focused on using both of these models in a generative way for concrete problems in cutting-edge scientific applications rather than the exclusive focus on the generation of high-resolution natural images. The research in this thesis is united around ideas of building models that can extract new knowledge from scientific data inaccessible to currently existing methods
Deep Decomposition Learning for Inverse Imaging Problems
Deep learning is emerging as a new paradigm for solving inverse imaging
problems. However, the deep learning methods often lack the assurance of
traditional physics-based methods due to the lack of physical information
considerations in neural network training and deploying. The appropriate
supervision and explicit calibration by the information of the physic model can
enhance the neural network learning and its practical performance. In this
paper, inspired by the geometry that data can be decomposed by two components
from the null-space of the forward operator and the range space of its
pseudo-inverse, we train neural networks to learn the two components and
therefore learn the decomposition, i.e. we explicitly reformulate the neural
network layers as learning range-nullspace decomposition functions with
reference to the layer inputs, instead of learning unreferenced functions. We
empirically show that the proposed framework demonstrates superior performance
over recent deep residual learning, unrolled learning and nullspace learning on
tasks including compressive sensing medical imaging and natural image
super-resolution. Our code is available at
https://github.com/edongdongchen/DDN.Comment: To appear in ECCV 202
- …