76 research outputs found
Restoring Vision in Hazy Weather with Hierarchical Contrastive Learning
Image restoration under hazy weather condition, which is called single image
dehazing, has been of significant interest for various computer vision
applications. In recent years, deep learning-based methods have achieved
success. However, existing image dehazing methods typically neglect the
hierarchy of features in the neural network and fail to exploit their
relationships fully. To this end, we propose an effective image dehazing method
named Hierarchical Contrastive Dehazing (HCD), which is based on feature fusion
and contrastive learning strategies. HCD consists of a hierarchical dehazing
network (HDN) and a novel hierarchical contrastive loss (HCL). Specifically,
the core design in the HDN is a hierarchical interaction module, which utilizes
multi-scale activation to revise the feature responses hierarchically. To
cooperate with the training of HDN, we propose HCL which performs contrastive
learning on hierarchically paired exemplars, facilitating haze removal.
Extensive experiments on public datasets, RESIDE, HazeRD, and DENSE-HAZE,
demonstrate that HCD quantitatively outperforms the state-of-the-art methods in
terms of PSNR, SSIM and achieves better visual quality.Comment: 30 pages, 10 figure
Learning Distortion Invariant Representation for Image Restoration from A Causality Perspective
In recent years, we have witnessed the great advancement of Deep neural
networks (DNNs) in image restoration. However, a critical limitation is that
they cannot generalize well to real-world degradations with different degrees
or types. In this paper, we are the first to propose a novel training strategy
for image restoration from the causality perspective, to improve the
generalization ability of DNNs for unknown degradations. Our method, termed
Distortion Invariant representation Learning (DIL), treats each distortion type
and degree as one specific confounder, and learns the distortion-invariant
representation by eliminating the harmful confounding effect of each
degradation. We derive our DIL with the back-door criterion in causality by
modeling the interventions of different distortions from the optimization
perspective. Particularly, we introduce counterfactual distortion augmentation
to simulate the virtual distortion types and degrees as the confounders. Then,
we instantiate the intervention of each distortion with a virtual model
updating based on corresponding distorted images, and eliminate them from the
meta-learning perspective. Extensive experiments demonstrate the effectiveness
of our DIL on the generalization capability for unseen distortion types and
degrees. Our code will be available at
https://github.com/lixinustc/Causal-IR-DIL.Comment: Accepted by CVPR202
Unsupervised Deraining: Where Contrastive Learning Meets Self-similarity
Image deraining is a typical low-level image restoration task, which aims at
decomposing the rainy image into two distinguishable layers: the clean image
layer and the rain layer. Most of the existing learning-based deraining methods
are supervisedly trained on synthetic rainy-clean pairs. The domain gap between
the synthetic and real rains makes them less generalized to different real
rainy scenes. Moreover, the existing methods mainly utilize the property of the
two layers independently, while few of them have considered the mutually
exclusive relationship between the two layers. In this work, we propose a novel
non-local contrastive learning (NLCL) method for unsupervised image deraining.
Consequently, we not only utilize the intrinsic self-similarity property within
samples but also the mutually exclusive property between the two layers, so as
to better differ the rain layer from the clean image. Specifically, the
non-local self-similarity image layer patches as the positives are pulled
together and similar rain layer patches as the negatives are pushed away. Thus
the similar positive/negative samples that are close in the original space
benefit us to enrich more discriminative representation. Apart from the
self-similarity sampling strategy, we analyze how to choose an appropriate
feature encoder in NLCL. Extensive experiments on different real rainy datasets
demonstrate that the proposed method obtains state-of-the-art performance in
real deraining.Comment: 10 pages, 10 figures, accept to 2022CVP
Video Adverse-Weather-Component Suppression Network via Weather Messenger and Adversarial Backpropagation
Although convolutional neural networks (CNNs) have been proposed to remove
adverse weather conditions in single images using a single set of pre-trained
weights, they fail to restore weather videos due to the absence of temporal
information. Furthermore, existing methods for removing adverse weather
conditions (e.g., rain, fog, and snow) from videos can only handle one type of
adverse weather. In this work, we propose the first framework for restoring
videos from all adverse weather conditions by developing a video
adverse-weather-component suppression network (ViWS-Net). To achieve this, we
first devise a weather-agnostic video transformer encoder with multiple
transformer stages. Moreover, we design a long short-term temporal modeling
mechanism for weather messenger to early fuse input adjacent video frames and
learn weather-specific information. We further introduce a weather
discriminator with gradient reversion, to maintain the weather-invariant common
information and suppress the weather-specific information in pixel features, by
adversarially predicting weather types. Finally, we develop a messenger-driven
video transformer decoder to retrieve the residual weather-specific feature,
which is spatiotemporally aggregated with hierarchical pixel features and
refined to predict the clean target frame of input videos. Experimental
results, on benchmark datasets and real-world weather videos, demonstrate that
our ViWS-Net outperforms current state-of-the-art methods in terms of restoring
videos degraded by any weather condition
Prompt-based All-in-One Image Restoration using CNNs and Transformer
Image restoration aims to recover the high-quality images from their degraded
observations. Since most existing methods have been dedicated into single
degradation removal, they may not yield optimal results on other types of
degradations, which do not satisfy the applications in real world scenarios. In
this paper, we propose a novel data ingredient-oriented approach that leverages
prompt-based learning to enable a single model to efficiently tackle multiple
image degradation tasks. Specifically, we utilize a encoder to capture features
and introduce prompts with degradation-specific information to guide the
decoder in adaptively recovering images affected by various degradations. In
order to model the local invariant properties and non-local information for
high-quality image restoration, we combined CNNs operations and Transformers.
Simultaneously, we made several key designs in the Transformer blocks
(multi-head rearranged attention with prompts and simple-gate feed-forward
network) to reduce computational requirements and selectively determines what
information should be persevered to facilitate efficient recovery of
potentially sharp images. Furthermore, we incorporate a feature fusion
mechanism further explores the multi-scale information to improve the
aggregated features. The resulting tightly interlinked hierarchy architecture,
named as CAPTNet, despite being designed to handle different types of
degradations, extensive experiments demonstrate that our method performs
competitively to the task-specific algorithms
Unsupervised Deraining: Where Asymmetric Contrastive Learning Meets Self-similarity
Most of the existing learning-based deraining methods are supervisedly
trained on synthetic rainy-clean pairs. The domain gap between the synthetic
and real rain makes them less generalized to complex real rainy scenes.
Moreover, the existing methods mainly utilize the property of the image or rain
layers independently, while few of them have considered their mutually
exclusive relationship. To solve above dilemma, we explore the intrinsic
intra-similarity within each layer and inter-exclusiveness between two layers
and propose an unsupervised non-local contrastive learning (NLCL) deraining
method. The non-local self-similarity image patches as the positives are
tightly pulled together, rain patches as the negatives are remarkably pushed
away, and vice versa. On one hand, the intrinsic self-similarity knowledge
within positive/negative samples of each layer benefits us to discover more
compact representation; on the other hand, the mutually exclusive property
between the two layers enriches the discriminative decomposition. Thus, the
internal self-similarity within each layer (similarity) and the external
exclusive relationship of the two layers (dissimilarity) serving as a generic
image prior jointly facilitate us to unsupervisedly differentiate the rain from
clean image. We further discover that the intrinsic dimension of the non-local
image patches is generally higher than that of the rain patches. This motivates
us to design an asymmetric contrastive loss to precisely model the compactness
discrepancy of the two layers for better discriminative decomposition. In
addition, considering that the existing real rain datasets are of low quality,
either small scale or downloaded from the internet, we collect a real
large-scale dataset under various rainy kinds of weather that contains
high-resolution rainy images.Comment: 16 pages, 15 figures. arXiv admin note: substantial text overlap with
arXiv:2203.1150
Joint Depth Estimation and Mixture of Rain Removal From a Single Image
Rainy weather significantly deteriorates the visibility of scene objects,
particularly when images are captured through outdoor camera lenses or
windshields. Through careful observation of numerous rainy photos, we have
found that the images are generally affected by various rainwater artifacts
such as raindrops, rain streaks, and rainy haze, which impact the image quality
from both near and far distances, resulting in a complex and intertwined
process of image degradation. However, current deraining techniques are limited
in their ability to address only one or two types of rainwater, which poses a
challenge in removing the mixture of rain (MOR). In this study, we propose an
effective image deraining paradigm for Mixture of rain REmoval, called
DEMore-Net, which takes full account of the MOR effect. Going beyond the
existing deraining wisdom, DEMore-Net is a joint learning paradigm that
integrates depth estimation and MOR removal tasks to achieve superior rain
removal. The depth information can offer additional meaningful guidance
information based on distance, thus better helping DEMore-Net remove different
types of rainwater. Moreover, this study explores normalization approaches in
image deraining tasks and introduces a new Hybrid Normalization Block (HNB) to
enhance the deraining performance of DEMore-Net. Extensive experiments
conducted on synthetic datasets and real-world MOR photos fully validate the
superiority of the proposed DEMore-Net. Code is available at
https://github.com/yz-wang/DEMore-Net.Comment: 11 pages, 7 figures, 5 table
- …