71 research outputs found
Video and Image Super-Resolution via Deep Learning with Attention Mechanism
Image demosaicing, image super-resolution and video super-resolution are three important tasks in color imaging pipeline. Demosaicing deals with the recovery of missing color information and generation of full-resolution color images from so-called Color filter Array (CFA) such as Bayer pattern. Image super-resolution aims at increasing the spatial resolution and enhance important structures (e.g., edges and textures) in super-resolved images. Both spatial and temporal dependency are important to the task of video super-resolution, which has received increasingly more attention in recent years. Traditional solutions to these three low-level vision tasks lack generalization capability especially for real-world data. Recently, deep learning methods have achieved great success in vision problems including image demosaicing and image/video super-resolution. Conceptually similar to adaptation in model-based approaches, attention has received increasing more usage in deep learning recently. As a tool to reallocate limited computational resources based on the importance of informative components, attention mechanism which includes channel attention, spatial attention, non-local attention, etc. has found successful applications in both highlevel and low-level vision tasks. However, to the best of our knowledge, 1) most approaches independently studied super-resolution and demosaicing; little is known about the potential benefit of formulating a joint demosaicing and super-resolution (JDSR) problem; 2) attention mechanism has not been studied for spectral channels of color images in the open literature; 3) current approaches for video super-resolution implement deformable convolution based frame alignment methods and naive spatial attention mechanism. How to exploit attention mechanism in spectral and temporal domains sets up the stage for the research in this dissertation. In this dissertation, we conduct a systematic study about those two issues and make the following contributions: 1) we propose a spatial color attention network (SCAN) designed to jointly exploit the spatial and spectral dependency within color images for single image super-resolution (SISR) problem. We present a spatial color attention module that calibrates important color information for individual color components from output feature maps of residual groups. Experimental results have shown that SCAN has achieved superior performance in terms of both subjective and objective qualities on the NTIRE2019 dataset; 2) we propose two competing end-to-end joint optimization solutions to the JDSR problem: Densely-Connected Squeeze-and-Excitation Residual Network (DSERN) vs. Residual-Dense Squeeze-and-Excitation Network (RDSEN). Experimental results have shown that an enhanced design RDSEN can significantly improve both subjective and objective performance over DSERN; 3) we propose a novel deep learning based framework, Deformable Kernel Spatial Attention Network (DKSAN) to super-resolve videos with a scale factor as large as 16 (the extreme SR situation). Thanks to newly designed Deformable Kernel Convolution Alignment (DKC Align) and Deformable Kernel Spatial Attention (DKSA) modules, DKSAN can get both better subjective and objective results when compared with the existing state-of-the-art approach enhanced deformable convolutional network (EDVR)
W2S: Microscopy Data with Joint Denoising and Super-Resolution for Widefield to SIM Mapping
In fluorescence microscopy live-cell imaging, there is a critical trade-off
between the signal-to-noise ratio and spatial resolution on one side, and the
integrity of the biological sample on the other side. To obtain clean
high-resolution (HR) images, one can either use microscopy techniques, such as
structured-illumination microscopy (SIM), or apply denoising and
super-resolution (SR) algorithms. However, the former option requires multiple
shots that can damage the samples, and although efficient deep learning based
algorithms exist for the latter option, no benchmark exists to evaluate these
algorithms on the joint denoising and SR (JDSR) tasks. To study JDSR on
microscopy data, we propose such a novel JDSR dataset, Widefield2SIM (W2S),
acquired using a conventional fluorescence widefield and SIM imaging. W2S
includes 144,000 real fluorescence microscopy images, resulting in a total of
360 sets of images. A set is comprised of noisy low-resolution (LR) widefield
images with different noise levels, a noise-free LR image, and a corresponding
high-quality HR SIM image. W2S allows us to benchmark the combinations of 6
denoising methods and 6 SR methods. We show that state-of-the-art SR networks
perform very poorly on noisy inputs. Our evaluation also reveals that applying
the best denoiser in terms of reconstruction error followed by the best SR
method does not necessarily yield the best final result. Both quantitative and
qualitative results show that SR networks are sensitive to noise and the
sequential application of denoising and SR algorithms is sub-optimal. Lastly,
we demonstrate that SR networks retrained end-to-end for JDSR outperform any
combination of state-of-the-art deep denoising and SR networksComment: ECCVW 2020. Project page: \<https://github.com/ivrl/w2s
Optimizing Image Compression via Joint Learning with Denoising
High levels of noise usually exist in today's captured images due to the
relatively small sensors equipped in the smartphone cameras, where the noise
brings extra challenges to lossy image compression algorithms. Without the
capacity to tell the difference between image details and noise, general image
compression methods allocate additional bits to explicitly store the undesired
image noise during compression and restore the unpleasant noisy image during
decompression. Based on the observations, we optimize the image compression
algorithm to be noise-aware as joint denoising and compression to resolve the
bits misallocation problem. The key is to transform the original noisy images
to noise-free bits by eliminating the undesired noise during compression, where
the bits are later decompressed as clean images. Specifically, we propose a
novel two-branch, weight-sharing architecture with plug-in feature denoisers to
allow a simple and effective realization of the goal with little computational
cost. Experimental results show that our method gains a significant improvement
over the existing baseline methods on both the synthetic and real-world
datasets. Our source code is available at
https://github.com/felixcheng97/DenoiseCompression.Comment: Accepted to ECCV 202
- …