24 research outputs found
Detail-recovery Image Deraining via Dual Sample-augmented Contrastive Learning
The intricacy of rainy image contents often leads cutting-edge deraining
models to image degradation including remnant rain, wrongly-removed details,
and distorted appearance. Such degradation is further exacerbated when applying
the models trained on synthetic data to real-world rainy images. We observe two
types of domain gaps between synthetic and real-world rainy images: one exists
in rain streak patterns; the other is the pixel-level appearance of rain-free
images. To bridge the two domain gaps, we propose a semi-supervised
detail-recovery image deraining network (Semi-DRDNet) with dual
sample-augmented contrastive learning. Semi-DRDNet consists of three
sub-networks:i) for removing rain streaks without remnants, we present a
squeeze-and-excitation based rain residual network; ii) for encouraging the
lost details to return, we construct a structure detail context aggregation
based detail repair network; to our knowledge, this is the first time; and iii)
for building efficient contrastive constraints for both rain streaks and clean
backgrounds, we exploit a novel dual sample-augmented contrastive
regularization network.Semi-DRDNet operates smoothly on both synthetic and
real-world rainy data in terms of deraining robustness and detail accuracy.
Comparisons on four datasets including our established Real200 show clear
improvements of Semi-DRDNet over fifteen state-of-the-art methods. Code and
dataset are available at https://github.com/syy-whu/DRD-Net.Comment: 17 page
μ¬μΈ΅ μ κ²½λ§μ μμ€νλ©΄ λ° λ₯λ¬λμ μ¬λ¬ μ μ©μ κ΄ν μ°κ΅¬
νμλ
Όλ¬Έ(λ°μ¬) -- μμΈλνκ΅λνμ : μμ°κ³Όνλν μ리과νλΆ, 2022. 8. κ°λͺ
μ£Ό.λ³Έ νμ λ
Όλ¬Έμ μ¬μΈ΅ μ κ²½λ§μ μμ€ νλ©΄μ λνμ¬ λ€λ£¬λ€. μ¬μΈ΅ μ κ²½λ§μ μμ€ ν¨μλ λ³Όλ‘ ν¨μμ κ°μ΄ λμ κ΅μμ μ κ°μ§λκ°? μ‘°κ°μ μΌλ‘ μ νμ νμ±ν¨μλ₯Ό κ°μ§λ κ²½μ°μ λν΄μλ μ μλ €μμ§λ§, μΌλ°μ μΈ λ§€λλ¬μ΄ νμ±ν¨μλ₯Ό κ°μ§λ μ¬μΈ΅ μ κ²½λ§μ λν΄μλ μμ§κΉμ§ μλ €μ§μ§ μμ κ²μ΄ λ§λ€. λ³Έ μ°κ΅¬μμλ λμ κ΅μμ μ΄ μΌλ°μ μΈ λ§€λλ¬μ΄ νμ±ν¨μμμλ μ‘΄μ¬ν¨μ 보μΈλ€. μ΄κ²μ μ¬μΈ΅ μ κ²½λ§μ μμ€ νλ©΄μ λν μ΄ν΄μ λΆλΆμ μΈ μ€λͺ
μ μ κ³΅ν΄ μ€ κ²μ΄λ€. μΆκ°μ μΌλ‘ λ³Έ λ
Όλ¬Έμμλ νμ΅ μ΄λ‘ , μ¬μν 보νΈμ μΈ κΈ°κ³ νμ΅, μ»΄ν¨ν° λΉμ λ±μ λΆμΌμμμ μ¬μΈ΅ μ κ²½λ§μ λ€μν μμ©μ μ λ³΄μΌ μμ μ΄λ€.In this thesis, we study the loss surface of deep neural networks. Does the loss function of deep neural network have no bad local minimum like the convex function? Although it is well known for piece-wise linear activations, not much is known for the general smooth activations. We explore that a bad local minimum also exists for general smooth activations. In addition, we characterize the types of such local minima. This provides a partial explanation for the understanding of the loss surface of deep neural networks. Additionally, we present several applications of deep neural networks in learning theory, private machine learning, and computer vision.Abstract v
1 Introduction 1
2 Existence of local minimum in neural network 4
2.1 Introduction 4
2.2 Local Minima and Deep Neural Network 6
2.2.1 Notation and Model 6
2.2.2 Local Minima and Deep Linear Network 6
2.2.3 Local Minima and Deep Neural Network with piece-wise linear activations 8
2.2.4 Local Minima and Deep Neural Network with smooth activations 10
2.2.5 Local Valley and Deep Neural Network 11
2.3 Existence of local minimum for partially linear activations 12
2.4 Absence of local minimum in the shallow network for small N 17
2.5 Existence of local minimum in the shallow network 20
2.6 Local Minimum Embedding 36
3 Self-Knowledge Distillation via Dropout 40
3.1 Introduction 40
3.2 Related work 43
3.2.1 Knowledge Distillation 43
3.2.2 Self-Knowledge Distillation 44
3.2.3 Semi-supervised and Self-supervised Learning 44
3.3 Self Distillation via Dropout 45
3.3.1 Method Formulation 46
3.3.2 Collaboration with other method 47
3.3.3 Forward versus reverse KL-Divergence 48
3.4 Experiments 53
3.4.1 Implementation Details 53
3.4.2 Results 54
3.5 Conclusion 62
4 Membership inference attacks against object detection models 63
4.1 Introduction 63
4.2 Background and Related Work 65
4.2.1 Membership Inference Attack 65
4.2.2 Object Detection 66
4.2.3 Datasets 67
4.3 Attack Methodology 67
4.3.1 Motivation 69
4.3.2 Gradient Tree Boosting 69
4.3.3 Convolutional Neural Network Based Method 70
4.3.4 Transfer Attack 73
4.4 Defense 73
4.4.1 Dropout 73
4.4.2 Diff erentially Private Algorithm 74
4.5 Experiments 75
4.5.1 Target and Shadow Model Setup 75
4.5.2 Attack Model Setup 77
4.5.3 Experiment Results 78
4.5.4 Transfer Attacks 80
4.5.5 Defense 81
4.6 Conclusion 81
5 Single Image Deraining 82
5.1 Introduction 82
5.2 Related Work 86
5.3 Proposed Network 89
5.3.1 Multi-Level Connection 89
5.3.2 Wide Regional Non-Local Block 92
5.3.3 Discrete Wavelet Transform 94
5.3.4 Loss Function 94
5.4 Experiments 95
5.4.1 Datasets and Evaluation Metrics 95
5.4.2 Datasets and Experiment Details 96
5.4.3 Evaluations 97
5.4.4 Ablation Study 104
5.4.5 Applications for Other Tasks 107
5.4.6 Analysis on multi-level features 109
5.5 Conclusion 111
The bibliography 112
Abstract (in Korean) 129λ°
Unsupervised Deraining: Where Contrastive Learning Meets Self-similarity
Image deraining is a typical low-level image restoration task, which aims at
decomposing the rainy image into two distinguishable layers: the clean image
layer and the rain layer. Most of the existing learning-based deraining methods
are supervisedly trained on synthetic rainy-clean pairs. The domain gap between
the synthetic and real rains makes them less generalized to different real
rainy scenes. Moreover, the existing methods mainly utilize the property of the
two layers independently, while few of them have considered the mutually
exclusive relationship between the two layers. In this work, we propose a novel
non-local contrastive learning (NLCL) method for unsupervised image deraining.
Consequently, we not only utilize the intrinsic self-similarity property within
samples but also the mutually exclusive property between the two layers, so as
to better differ the rain layer from the clean image. Specifically, the
non-local self-similarity image layer patches as the positives are pulled
together and similar rain layer patches as the negatives are pushed away. Thus
the similar positive/negative samples that are close in the original space
benefit us to enrich more discriminative representation. Apart from the
self-similarity sampling strategy, we analyze how to choose an appropriate
feature encoder in NLCL. Extensive experiments on different real rainy datasets
demonstrate that the proposed method obtains state-of-the-art performance in
real deraining.Comment: 10 pages, 10 figures, accept to 2022CVP
RCDNet: An Interpretable Rain Convolutional Dictionary Network for Single Image Deraining
As a common weather, rain streaks adversely degrade the image quality. Hence,
removing rains from an image has become an important issue in the field. To
handle such an ill-posed single image deraining task, in this paper, we
specifically build a novel deep architecture, called rain convolutional
dictionary network (RCDNet), which embeds the intrinsic priors of rain streaks
and has clear interpretability. In specific, we first establish a RCD model for
representing rain streaks and utilize the proximal gradient descent technique
to design an iterative algorithm only containing simple operators for solving
the model. By unfolding it, we then build the RCDNet in which every network
module has clear physical meanings and corresponds to each operation involved
in the algorithm. This good interpretability greatly facilitates an easy
visualization and analysis on what happens inside the network and why it works
well in inference process. Moreover, taking into account the domain gap issue
in real scenarios, we further design a novel dynamic RCDNet, where the rain
kernels can be dynamically inferred corresponding to input rainy images and
then help shrink the space for rain layer estimation with few rain maps so as
to ensure a fine generalization performance in the inconsistent scenarios of
rain types between training and testing data. By end-to-end training such an
interpretable network, all involved rain kernels and proximal operators can be
automatically extracted, faithfully characterizing the features of both rain
and clean background layers, and thus naturally lead to better deraining
performance. Comprehensive experiments substantiate the superiority of our
method, especially on its well generality to diverse testing scenarios and good
interpretability for all its modules. Code is available in
\emph{\url{https://github.com/hongwang01/DRCDNet}}
Unsupervised Deraining: Where Asymmetric Contrastive Learning Meets Self-similarity
Most of the existing learning-based deraining methods are supervisedly
trained on synthetic rainy-clean pairs. The domain gap between the synthetic
and real rain makes them less generalized to complex real rainy scenes.
Moreover, the existing methods mainly utilize the property of the image or rain
layers independently, while few of them have considered their mutually
exclusive relationship. To solve above dilemma, we explore the intrinsic
intra-similarity within each layer and inter-exclusiveness between two layers
and propose an unsupervised non-local contrastive learning (NLCL) deraining
method. The non-local self-similarity image patches as the positives are
tightly pulled together, rain patches as the negatives are remarkably pushed
away, and vice versa. On one hand, the intrinsic self-similarity knowledge
within positive/negative samples of each layer benefits us to discover more
compact representation; on the other hand, the mutually exclusive property
between the two layers enriches the discriminative decomposition. Thus, the
internal self-similarity within each layer (similarity) and the external
exclusive relationship of the two layers (dissimilarity) serving as a generic
image prior jointly facilitate us to unsupervisedly differentiate the rain from
clean image. We further discover that the intrinsic dimension of the non-local
image patches is generally higher than that of the rain patches. This motivates
us to design an asymmetric contrastive loss to precisely model the compactness
discrepancy of the two layers for better discriminative decomposition. In
addition, considering that the existing real rain datasets are of low quality,
either small scale or downloaded from the internet, we collect a real
large-scale dataset under various rainy kinds of weather that contains
high-resolution rainy images.Comment: 16 pages, 15 figures. arXiv admin note: substantial text overlap with
arXiv:2203.1150
Toward Real-world Single Image Deraining: A New Benchmark and Beyond
Single image deraining (SID) in real scenarios attracts increasing attention
in recent years. Due to the difficulty in obtaining real-world rainy/clean
image pairs, previous real datasets suffer from low-resolution images,
homogeneous rain streaks, limited background variation, and even misalignment
of image pairs, resulting in incomprehensive evaluation of SID methods. To
address these issues, we establish a new high-quality dataset named
RealRain-1k, consisting of high-resolution paired clean and rainy
images with low- and high-density rain streaks, respectively. Images in
RealRain-1k are automatically generated from a large number of real-world rainy
video clips through a simple yet effective rain density-controllable filtering
method, and have good properties of high image resolution, background
diversity, rain streaks variety, and strict spatial alignment. RealRain-1k also
provides abundant rain streak layers as a byproduct, enabling us to build a
large-scale synthetic dataset named SynRain-13k by pasting the rain streak
layers on abundant natural images. Based on them and existing datasets, we
benchmark more than 10 representative SID methods on three tracks: (1) fully
supervised learning on RealRain-1k, (2) domain generalization to real datasets,
and (3) syn-to-real transfer learning. The experimental results (1) show the
difference of representative methods in image restoration performance and model
complexity, (2) validate the significance of the proposed datasets for model
generalization, and (3) provide useful insights on the superiority of learning
from diverse domains and shed lights on the future research on real-world SID.
The datasets will be released at https://github.com/hiker-lw/RealRain-1
Blind Image Decomposition
We propose and study a novel task named Blind Image Decomposition (BID),
which requires separating a superimposed image into constituent underlying
images in a blind setting, that is, both the source components involved in
mixing as well as the mixing mechanism are unknown. For example, rain may
consist of multiple components, such as rain streaks, raindrops, snow, and
haze. Rainy images can be treated as an arbitrary combination of these
components, some of them or all of them. How to decompose superimposed images,
like rainy images, into distinct source components is a crucial step toward
real-world vision systems. To facilitate research on this new task, we
construct multiple benchmark datasets, including mixed image decomposition
across multiple domains, real-scenario deraining, and joint
shadow/reflection/watermark removal. Moreover, we propose a simple yet general
Blind Image Decomposition Network (BIDeN) to serve as a strong baseline for
future work. Experimental results demonstrate the tenability of our benchmarks
and the effectiveness of BIDeN.Comment: ECCV 2022. Project page:
https://junlinhan.github.io/projects/BID.html. Code:
https://github.com/JunlinHan/BI