3,910 research outputs found
Divide-and-conquer framework for image restoration and enhancement
Abstract(#br)We develop a novel divide-and-conquer framework for image restoration and enhancement based on their task-driven requirements, which takes advantage of visual importance differences of image contents (i.e., noise versus image, edge-based structures versus smoothing areas, high-frequency versus low-frequency components) and sparse prior differences of image contents for performance improvements. The proposed framework is efficient in implementation of decomposition-processing-integration. An observed image is first decomposed into different subspaces based on considering visual importance of different subspaces and exploiting their prior differences. Different models are separately established for image subspace restoration and enhancement, and existing image restoration and enhancement methods are utilized to deal with them effectively. Then a simple but effective fusion scheme with different weights is used to integrate the post-processed subspaces for the final reconstructed image. Final experimental results demonstrate that the proposed divide-and-conquer framework outperforms several restoration and enhancement algorithms in both subjective results and objective assessments. The performance improvements of image restoration and enhancement can be yielded by using the proposed divide-and-conquer strategy, which greatly benefits in terms of mixed Gaussian and salt-and-pepper noise removal, non-blind deconvolution, and image enhancement. In addition, our divide-and-conquer framework can be simply extensible to other restoration and enhancement algorithms, and can be a new way to promote their performances for image restoration and enhancement
JSI-GAN: GAN-Based Joint Super-Resolution and Inverse Tone-Mapping with Pixel-Wise Task-Specific Filters for UHD HDR Video
Joint learning of super-resolution (SR) and inverse tone-mapping (ITM) has
been explored recently, to convert legacy low resolution (LR) standard dynamic
range (SDR) videos to high resolution (HR) high dynamic range (HDR) videos for
the growing need of UHD HDR TV/broadcasting applications. However, previous
CNN-based methods directly reconstruct the HR HDR frames from LR SDR frames,
and are only trained with a simple L2 loss. In this paper, we take a
divide-and-conquer approach in designing a novel GAN-based joint SR-ITM
network, called JSI-GAN, which is composed of three task-specific subnets: an
image reconstruction subnet, a detail restoration (DR) subnet and a local
contrast enhancement (LCE) subnet. We delicately design these subnets so that
they are appropriately trained for the intended purpose, learning a pair of
pixel-wise 1D separable filters via the DR subnet for detail restoration and a
pixel-wise 2D local filter by the LCE subnet for contrast enhancement.
Moreover, to train the JSI-GAN effectively, we propose a novel detail GAN loss
alongside the conventional GAN loss, which helps enhancing both local details
and contrasts to reconstruct high quality HR HDR results. When all subnets are
jointly trained well, the predicted HR HDR results of higher quality are
obtained with at least 0.41 dB gain in PSNR over those generated by the
previous methods.Comment: The first two authors contributed equally to this work. Accepted at
AAAI 2020. (Camera-ready version
Mutual Guidance and Residual Integration for Image Enhancement
Previous studies show the necessity of global and local adjustment for image
enhancement. However, existing convolutional neural networks (CNNs) and
transformer-based models face great challenges in balancing the computational
efficiency and effectiveness of global-local information usage. Especially,
existing methods typically adopt the global-to-local fusion mode, ignoring the
importance of bidirectional interactions. To address those issues, we propose a
novel mutual guidance network (MGN) to perform effective bidirectional
global-local information exchange while keeping a compact architecture. In our
design, we adopt a two-branch framework where one branch focuses more on
modeling global relations while the other is committed to processing local
information. Then, we develop an efficient attention-based mutual guidance
approach throughout our framework for bidirectional global-local interactions.
As a result, both the global and local branches can enjoy the merits of mutual
information aggregation. Besides, to further refine the results produced by our
MGN, we propose a novel residual integration scheme following the
divide-and-conquer philosophy. The extensive experiments demonstrate the
effectiveness of our proposed method, which achieves state-of-the-art
performance on several public image enhancement benchmarks.Comment: 17 pages, 15 figure
Division Gets Better: Learning Brightness-Aware and Detail-Sensitive Representations for Low-Light Image Enhancement
Low-light image enhancement strives to improve the contrast, adjust the
visibility, and restore the distortion in color and texture. Existing methods
usually pay more attention to improving the visibility and contrast via
increasing the lightness of low-light images, while disregarding the
significance of color and texture restoration for high-quality images. Against
above issue, we propose a novel luminance and chrominance dual branch network,
termed LCDBNet, for low-light image enhancement, which divides low-light image
enhancement into two sub-tasks, e.g., luminance adjustment and chrominance
restoration. Specifically, LCDBNet is composed of two branches, namely
luminance adjustment network (LAN) and chrominance restoration network (CRN).
LAN takes responsibility for learning brightness-aware features leveraging
long-range dependency and local attention correlation. While CRN concentrates
on learning detail-sensitive features via multi-level wavelet decomposition.
Finally, a fusion network is designed to blend their learned features to
produce visually impressive images. Extensive experiments conducted on seven
benchmark datasets validate the effectiveness of our proposed LCDBNet, and the
results manifest that LCDBNet achieves superior performance in terms of
multiple reference/non-reference quality evaluators compared to other
state-of-the-art competitors. Our code and pretrained model will be available.Comment: 14 pages, 16 figure
Learning Disentangled Feature Representation for Hybrid-distorted Image Restoration
Hybrid-distorted image restoration (HD-IR) is dedicated to restore real
distorted image that is degraded by multiple distortions. Existing HD-IR
approaches usually ignore the inherent interference among hybrid distortions
which compromises the restoration performance. To decompose such interference,
we introduce the concept of Disentangled Feature Learning to achieve the
feature-level divide-and-conquer of hybrid distortions. Specifically, we
propose the feature disentanglement module (FDM) to distribute feature
representations of different distortions into different channels by revising
gain-control-based normalization. We also propose a feature aggregation module
(FAM) with channel-wise attention to adaptively filter out the distortion
representations and aggregate useful content information from different
channels for the construction of raw image. The effectiveness of the proposed
scheme is verified by visualizing the correlation matrix of features and
channel responses of different distortions. Extensive experimental results also
prove superior performance of our approach compared with the latest HD-IR
schemes.Comment: Accepted by ECCV202
Physics-Driven Turbulence Image Restoration with Stochastic Refinement
Image distortion by atmospheric turbulence is a stochastic degradation, which
is a critical problem in long-range optical imaging systems. A number of
research has been conducted during the past decades, including model-based and
emerging deep-learning solutions with the help of synthetic data. Although fast
and physics-grounded simulation tools have been introduced to help the
deep-learning models adapt to real-world turbulence conditions recently, the
training of such models only relies on the synthetic data and ground truth
pairs. This paper proposes the Physics-integrated Restoration Network (PiRN) to
bring the physics-based simulator directly into the training process to help
the network to disentangle the stochasticity from the degradation and the
underlying image. Furthermore, to overcome the ``average effect" introduced by
deterministic models and the domain gap between the synthetic and real-world
degradation, we further introduce PiRN with Stochastic Refinement (PiRN-SR) to
boost its perceptual quality. Overall, our PiRN and PiRN-SR improve the
generalization to real-world unknown turbulence conditions and provide a
state-of-the-art restoration in both pixel-wise accuracy and perceptual
quality. Our codes are available at \url{https://github.com/VITA-Group/PiRN}.Comment: Accepted by ICCV 202
Learning Image-Adaptive Codebooks for Class-Agnostic Image Restoration
Recent work on discrete generative priors, in the form of codebooks, has
shown exciting performance for image reconstruction and restoration, as the
discrete prior space spanned by the codebooks increases the robustness against
diverse image degradations. Nevertheless, these methods require separate
training of codebooks for different image categories, which limits their use to
specific image categories only (e.g. face, architecture, etc.), and fail to
handle arbitrary natural images. In this paper, we propose AdaCode for learning
image-adaptive codebooks for class-agnostic image restoration. Instead of
learning a single codebook for each image category, we learn a set of basis
codebooks. For a given input image, AdaCode learns a weight map with which we
compute a weighted combination of these basis codebooks for adaptive image
restoration. Intuitively, AdaCode is a more flexible and expressive discrete
generative prior than previous work. Experimental results demonstrate that
AdaCode achieves state-of-the-art performance on image reconstruction and
restoration tasks, including image super-resolution and inpainting
- …