687 research outputs found
Learning Disentangled Feature Representation for Hybrid-distorted Image Restoration
Hybrid-distorted image restoration (HD-IR) is dedicated to restore real
distorted image that is degraded by multiple distortions. Existing HD-IR
approaches usually ignore the inherent interference among hybrid distortions
which compromises the restoration performance. To decompose such interference,
we introduce the concept of Disentangled Feature Learning to achieve the
feature-level divide-and-conquer of hybrid distortions. Specifically, we
propose the feature disentanglement module (FDM) to distribute feature
representations of different distortions into different channels by revising
gain-control-based normalization. We also propose a feature aggregation module
(FAM) with channel-wise attention to adaptively filter out the distortion
representations and aggregate useful content information from different
channels for the construction of raw image. The effectiveness of the proposed
scheme is verified by visualizing the correlation matrix of features and
channel responses of different distortions. Extensive experimental results also
prove superior performance of our approach compared with the latest HD-IR
schemes.Comment: Accepted by ECCV202
Learning Distortion Invariant Representation for Image Restoration from A Causality Perspective
In recent years, we have witnessed the great advancement of Deep neural
networks (DNNs) in image restoration. However, a critical limitation is that
they cannot generalize well to real-world degradations with different degrees
or types. In this paper, we are the first to propose a novel training strategy
for image restoration from the causality perspective, to improve the
generalization ability of DNNs for unknown degradations. Our method, termed
Distortion Invariant representation Learning (DIL), treats each distortion type
and degree as one specific confounder, and learns the distortion-invariant
representation by eliminating the harmful confounding effect of each
degradation. We derive our DIL with the back-door criterion in causality by
modeling the interventions of different distortions from the optimization
perspective. Particularly, we introduce counterfactual distortion augmentation
to simulate the virtual distortion types and degrees as the confounders. Then,
we instantiate the intervention of each distortion with a virtual model
updating based on corresponding distorted images, and eliminate them from the
meta-learning perspective. Extensive experiments demonstrate the effectiveness
of our DIL on the generalization capability for unseen distortion types and
degrees. Our code will be available at
https://github.com/lixinustc/Causal-IR-DIL.Comment: Accepted by CVPR202
Hybrid Neural Rendering for Large-Scale Scenes with Motion Blur
Rendering novel view images is highly desirable for many applications.
Despite recent progress, it remains challenging to render high-fidelity and
view-consistent novel views of large-scale scenes from in-the-wild images with
inevitable artifacts (e.g., motion blur). To this end, we develop a hybrid
neural rendering model that makes image-based representation and neural 3D
representation join forces to render high-quality, view-consistent images.
Besides, images captured in the wild inevitably contain artifacts, such as
motion blur, which deteriorates the quality of rendered images. Accordingly, we
propose strategies to simulate blur effects on the rendered images to mitigate
the negative influence of blurriness images and reduce their importance during
training based on precomputed quality-aware weights. Extensive experiments on
real and synthetic data demonstrate our model surpasses state-of-the-art
point-based methods for novel view synthesis. The code is available at
https://daipengwa.github.io/Hybrid-Rendering-ProjectPage
Multi-view Self-supervised Disentanglement for General Image Denoising
With its significant performance improvements, the deep learning paradigm has
become a standard tool for modern image denoisers. While promising performance
has been shown on seen noise distributions, existing approaches often suffer
from generalisation to unseen noise types or general and real noise. It is
understandable as the model is designed to learn paired mapping (e.g. from a
noisy image to its clean version). In this paper, we instead propose to learn
to disentangle the noisy image, under the intuitive assumption that different
corrupted versions of the same clean image share a common latent space. A
self-supervised learning framework is proposed to achieve the goal, without
looking at the latent clean image. By taking two different corrupted versions
of the same image as input, the proposed Multi-view Self-supervised
Disentanglement (MeD) approach learns to disentangle the latent clean features
from the corruptions and recover the clean image consequently. Extensive
experimental analysis on both synthetic and real noise shows the superiority of
the proposed method over prior self-supervised approaches, especially on unseen
novel noise types. On real noise, the proposed method even outperforms its
supervised counterparts by over 3 dB.Comment: International Conference on Computer Vision 2023 (ICCV 2023
A Dive into SAM Prior in Image Restoration
The goal of image restoration (IR), a fundamental issue in computer vision,
is to restore a high-quality (HQ) image from its degraded low-quality (LQ)
observation. Multiple HQ solutions may correspond to an LQ input in this poorly
posed problem, creating an ambiguous solution space. This motivates the
investigation and incorporation of prior knowledge in order to effectively
constrain the solution space and enhance the quality of the restored images. In
spite of the pervasive use of hand-crafted and learned priors in IR, limited
attention has been paid to the incorporation of knowledge from large-scale
foundation models. In this paper, we for the first time leverage the prior
knowledge of the state-of-the-art segment anything model (SAM) to boost the
performance of existing IR networks in an parameter-efficient tuning manner. In
particular, the choice of SAM is based on its robustness to image degradations,
such that HQ semantic masks can be extracted from it. In order to leverage
semantic priors and enhance restoration quality, we propose a lightweight SAM
prior tuning (SPT) unit. This plug-and-play component allows us to effectively
integrate semantic priors into existing IR networks, resulting in significant
improvements in restoration quality. As the only trainable module in our
method, the SPT unit has the potential to improve both efficiency and
scalability. We demonstrate the effectiveness of the proposed method in
enhancing a variety of methods across multiple tasks, such as image
super-resolution and color image denoising.Comment: Technical Repor
Prompt-based All-in-One Image Restoration using CNNs and Transformer
Image restoration aims to recover the high-quality images from their degraded
observations. Since most existing methods have been dedicated into single
degradation removal, they may not yield optimal results on other types of
degradations, which do not satisfy the applications in real world scenarios. In
this paper, we propose a novel data ingredient-oriented approach that leverages
prompt-based learning to enable a single model to efficiently tackle multiple
image degradation tasks. Specifically, we utilize a encoder to capture features
and introduce prompts with degradation-specific information to guide the
decoder in adaptively recovering images affected by various degradations. In
order to model the local invariant properties and non-local information for
high-quality image restoration, we combined CNNs operations and Transformers.
Simultaneously, we made several key designs in the Transformer blocks
(multi-head rearranged attention with prompts and simple-gate feed-forward
network) to reduce computational requirements and selectively determines what
information should be persevered to facilitate efficient recovery of
potentially sharp images. Furthermore, we incorporate a feature fusion
mechanism further explores the multi-scale information to improve the
aggregated features. The resulting tightly interlinked hierarchy architecture,
named as CAPTNet, despite being designed to handle different types of
degradations, extensive experiments demonstrate that our method performs
competitively to the task-specific algorithms
- …