12 research outputs found
Fast Saturating Gate for Learning Long Time Scales with Recurrent Neural Networks
Gate functions in recurrent models, such as an LSTM and GRU, play a central
role in learning various time scales in modeling time series data by using a
bounded activation function. However, it is difficult to train gates to capture
extremely long time scales due to gradient vanishing of the bounded function
for large inputs, which is known as the saturation problem. We closely analyze
the relation between saturation of the gate function and efficiency of the
training. We prove that the gradient vanishing of the gate function can be
mitigated by accelerating the convergence of the saturating function, i.e.,
making the output of the function converge to 0 or 1 faster. Based on the
analysis results, we propose a gate function called fast gate that has a doubly
exponential convergence rate with respect to inputs by simple function
composition. We empirically show that our method outperforms previous methods
in accuracy and computational efficiency on benchmark tasks involving extremely
long time scales.Comment: 9 pages of main texts with 4 pages appendices, 12 figure
Effective Data Augmentation with Multi-Domain Learning GANs
For deep learning applications, the massive data development (e.g.,
collecting, labeling), which is an essential process in building practical
applications, still incurs seriously high costs. In this work, we propose an
effective data augmentation method based on generative adversarial networks
(GANs), called Domain Fusion. Our key idea is to import the knowledge contained
in an outer dataset to a target model by using a multi-domain learning GAN. The
multi-domain learning GAN simultaneously learns the outer and target dataset
and generates new samples for the target tasks. The simultaneous learning
process makes GANs generate the target samples with high fidelity and variety.
As a result, we can obtain accurate models for the target tasks by using these
generated samples even if we only have an extremely low volume target dataset.
We experimentally evaluate the advantages of Domain Fusion in image
classification tasks on 3 target datasets: CIFAR-100, FGVC-Aircraft, and Indoor
Scene Recognition. When trained on each target dataset reduced the samples to
5,000 images, Domain Fusion achieves better classification accuracy than the
data augmentation using fine-tuned GANs. Furthermore, we show that Domain
Fusion improves the quality of generated samples, and the improvements can
contribute to higher accuracy.Comment: AAAI-202
Regularizing Neural Networks with Meta-Learning Generative Models
This paper investigates methods for improving generative data augmentation
for deep learning. Generative data augmentation leverages the synthetic samples
produced by generative models as an additional dataset for classification with
small dataset settings. A key challenge of generative data augmentation is that
the synthetic data contain uninformative samples that degrade accuracy. This is
because the synthetic samples do not perfectly represent class categories in
real data and uniform sampling does not necessarily provide useful samples for
tasks. In this paper, we present a novel strategy for generative data
augmentation called meta generative regularization (MGR). To avoid the
degradation of generative data augmentation, MGR utilizes synthetic samples in
the regularization term for feature extractors instead of in the loss function,
e.g., cross-entropy. These synthetic samples are dynamically determined to
minimize the validation losses through meta-learning. We observed that MGR can
avoid the performance degradation of na\"ive generative data augmentation and
boost the baselines. Experiments on six datasets showed that MGR is effective
particularly when datasets are smaller and stably outperforms baselines.Comment: Accepted to NeurIPS 202
Transfer Learning with Pre-trained Conditional Generative Models
Transfer learning is crucial in training deep neural networks on new target
tasks. Current transfer learning methods always assume at least one of (i)
source and target task label spaces overlap, (ii) source datasets are
available, and (iii) target network architectures are consistent with source
ones. However, holding these assumptions is difficult in practical settings
because the target task rarely has the same labels as the source task, the
source dataset access is restricted due to storage costs and privacy, and the
target architecture is often specialized to each task. To transfer source
knowledge without these assumptions, we propose a transfer learning method that
uses deep generative models and is composed of the following two stages: pseudo
pre-training (PP) and pseudo semi-supervised learning (P-SSL). PP trains a
target architecture with an artificial dataset synthesized by using conditional
source generative models. P-SSL applies SSL algorithms to labeled target data
and unlabeled pseudo samples, which are generated by cascading the source
classifier and generative models to condition them with target samples. Our
experimental results indicate that our method can outperform the baselines of
scratch training and knowledge distillation.Comment: 24 pages, 6 figure
Smoothness Analysis of Adversarial Training
Deep neural networks are vulnerable to adversarial attacks. Recent studies
about adversarial robustness focus on the loss landscape in the parameter space
since it is related to optimization and generalization performance. These
studies conclude that the difficulty of adversarial training is caused by the
non-smoothness of the loss function: i.e., its gradient is not Lipschitz
continuous. However, this analysis ignores the dependence of adversarial
attacks on model parameters. Since adversarial attacks are optimized for
models, they should depend on the parameters. Considering this dependence, we
analyze the smoothness of the loss function of adversarial training using the
optimal attacks for the model parameter in more detail. We reveal that the
constraint of adversarial attacks is one cause of the non-smoothness and that
the smoothness depends on the types of the constraints. Specifically, the
constraint can cause non-smoothness more than the constraint.
Moreover, our analysis implies that if we flatten the loss function with
respect to input data, the Lipschitz constant of the gradient of adversarial
loss tends to increase. To address the non-smoothness, we show that EntropySGD
smoothens the non-smooth loss and improves the performance of adversarial
training.Comment: 22 pages, 7 figures. In V3, we add the results of EntropySGD for
adversarial trainin
Switching One-Versus-the-Rest Loss to Increase the Margin of Logits for Adversarial Robustness
Adversarial training is a promising method to improve the robustness against
adversarial attacks. To enhance its performance, recent methods impose high
weights on the cross-entropy loss for important data points near the decision
boundary. However, these importance-aware methods are vulnerable to
sophisticated attacks, e.g., Auto-Attack. In this paper, we experimentally
investigate the cause of their vulnerability via margins between logits for the
true label and the other labels because they should be large enough to prevent
the largest logit from being flipped by the attacks. Our experiments reveal
that the histogram of the logit margins of na\"ive adversarial training has two
peaks. Thus, the levels of difficulty in increasing logit margins are roughly
divided into two: difficult samples (small logit margins) and easy samples
(large logit margins). On the other hand, only one peak near zero appears in
the histogram of importance-aware methods, i.e., they reduce the logit margins
of easy samples. To increase logit margins of difficult samples without
reducing those of easy samples, we propose switching one-versus-the-rest loss
(SOVR), which switches from cross-entropy to one-versus-the-rest loss (OVR) for
difficult samples. We derive trajectories of logit margins for a simple problem
and prove that OVR increases logit margins two times larger than the weighted
cross-entropy loss. Thus, SOVR increases logit margins of difficult samples,
unlike existing methods. We experimentally show that SOVR achieves better
robustness against Auto-Attack than importance-aware methods.Comment: 25 pages, 18 figure
Adversarial Finetuning with Latent Representation Constraint to Mitigate Accuracy-Robustness Tradeoff
This paper addresses the tradeoff between standard accuracy on clean examples
and robustness against adversarial examples in deep neural networks (DNNs).
Although adversarial training (AT) improves robustness, it degrades the
standard accuracy, thus yielding the tradeoff. To mitigate this tradeoff, we
propose a novel AT method called ARREST, which comprises three components: (i)
adversarial finetuning (AFT), (ii) representation-guided knowledge distillation
(RGKD), and (iii) noisy replay (NR). AFT trains a DNN on adversarial examples
by initializing its parameters with a DNN that is standardly pretrained on
clean examples. RGKD and NR respectively entail a regularization term and an
algorithm to preserve latent representations of clean examples during AFT. RGKD
penalizes the distance between the representations of the standardly pretrained
and AFT DNNs. NR switches input adversarial examples to nonadversarial ones
when the representation changes significantly during AFT. By combining these
components, ARREST achieves both high standard accuracy and robustness.
Experimental results demonstrate that ARREST mitigates the tradeoff more
effectively than previous AT-based methods do.Comment: Accepted by International Conference on Computer Vision (ICCV) 202
Absum: Simple Regularization Method for Reducing Structural Sensitivity of Convolutional Neural Networks
We propose Absum, which is a regularization method for improving adversarial
robustness of convolutional neural networks (CNNs). Although CNNs can
accurately recognize images, recent studies have shown that the convolution
operations in CNNs commonly have structural sensitivity to specific noise
composed of Fourier basis functions. By exploiting this sensitivity, they
proposed a simple black-box adversarial attack: Single Fourier attack. To
reduce structural sensitivity, we can use regularization of convolution filter
weights since the sensitivity of linear transform can be assessed by the norm
of the weights. However, standard regularization methods can prevent
minimization of the loss function because they impose a tight constraint for
obtaining high robustness. To solve this problem, Absum imposes a loose
constraint; it penalizes the absolute values of the summation of the parameters
in the convolution layers. Absum can improve robustness against single Fourier
attack while being as simple and efficient as standard regularization methods
(e.g., weight decay and L1 regularization). Our experiments demonstrate that
Absum improves robustness against single Fourier attack more than standard
regularization methods. Furthermore, we reveal that robust CNNs with Absum are
more robust against transferred attacks due to decreasing the common
sensitivity and against high-frequency noise than standard regularization
methods. We also reveal that Absum can improve robustness against
gradient-based attacks (projected gradient descent) when used with adversarial
training.Comment: 16 pages, 39 figure