15 research outputs found
Sparsity Meets Robustness: Channel Pruning for the Feynman-Kac Formalism Principled Robust Deep Neural Nets
Deep neural nets (DNNs) compression is crucial for adaptation to mobile
devices. Though many successful algorithms exist to compress naturally trained
DNNs, developing efficient and stable compression algorithms for robustly
trained DNNs remains widely open. In this paper, we focus on a co-design of
efficient DNN compression algorithms and sparse neural architectures for robust
and accurate deep learning. Such a co-design enables us to advance the goal of
accommodating both sparsity and robustness. With this objective in mind, we
leverage the relaxed augmented Lagrangian based algorithms to prune the weights
of adversarially trained DNNs, at both structured and unstructured levels.
Using a Feynman-Kac formalism principled robust and sparse DNNs, we can at
least double the channel sparsity of the adversarially trained ResNet20 for
CIFAR10 classification, meanwhile, improve the natural accuracy by \% and
the robust accuracy under the benchmark iterations of IFGSM attack by
\%. The code is available at
\url{https://github.com/BaoWangMath/rvsm-rgsm-admm}.Comment: 16 pages, 7 figure
An Integrated Approach to Produce Robust Models with High Efficiency
Deep Neural Networks (DNNs) needs to be both efficient and robust for
practical uses. Quantization and structure simplification are promising ways to
adapt DNNs to mobile devices, and adversarial training is the most popular
method to make DNNs robust. In this work, we try to obtain both features by
applying a convergent relaxation quantization algorithm, Binary-Relax (BR), to
a robust adversarial-trained model, ResNets Ensemble via Feynman-Kac Formalism
(EnResNet). We also discover that high precision, such as ternary (tnn) and
4-bit, quantization will produce sparse DNNs. However, this sparsity is
unstructured under advarsarial training. To solve the problems that adversarial
training jeopardizes DNNs' accuracy on clean images and the struture of
sparsity, we design a trade-off loss function that helps DNNs preserve their
natural accuracy and improve the channel sparsity. With our trade-off loss
function, we achieve both goals with no reduction of resistance under weak
attacks and very minor reduction of resistance under strong attcks. Together
with quantized EnResNet with trade-off loss function, we provide robust models
that have high efficiency
Achieving Adversarial Robustness via Sparsity
Network pruning has been known to produce compact models without much
accuracy degradation. However, how the pruning process affects a network's
robustness and the working mechanism behind remain unresolved. In this work, we
theoretically prove that the sparsity of network weights is closely associated
with model robustness. Through experiments on a variety of adversarial pruning
methods, we find that weights sparsity will not hurt but improve robustness,
where both weights inheritance from the lottery ticket and adversarial training
improve model robustness in network pruning. Based on these findings, we
propose a novel adversarial training method called inverse weights inheritance,
which imposes sparse weights distribution on a large network by inheriting
weights from a small network, thereby improving the robustness of the large
network
STEER: Simple Temporal Regularization For Neural ODEs
Training Neural Ordinary Differential Equations (ODEs) is often
computationally expensive. Indeed, computing the forward pass of such models
involves solving an ODE which can become arbitrarily complex during training.
Recent works have shown that regularizing the dynamics of the ODE can partially
alleviate this. In this paper we propose a new regularization technique:
randomly sampling the end time of the ODE during training. The proposed
regularization is simple to implement, has negligible overhead and is effective
across a wide variety of tasks. Further, the technique is orthogonal to several
other methods proposed to regularize the dynamics of ODEs and as such can be
used in conjunction with them. We show through experiments on normalizing
flows, time series models and image recognition that the proposed
regularization can significantly decrease training time and even improve
performance over baseline models.Comment: Neurips 202
Towards Optimal Randomized Strategies in Adversarial Example Game
The vulnerability of deep neural network models to adversarial example
attacks is a practical challenge in many artificial intelligence applications.
A recent line of work shows that the use of randomization in adversarial
training is the key to find optimal strategies against adversarial example
attacks. However, in a fully randomized setting where both the defender and the
attacker can use randomized strategies, there are no efficient algorithm for
finding such an optimal strategy. To fill the gap, we propose the first
algorithm of its kind, called FRAT, which models the problem with a new
infinite-dimensional continuous-time flow on probability distribution spaces.
FRAT maintains a lightweight mixture of models for the defender, with
flexibility to efficiently update mixing weights and model parameters at each
iteration. Furthermore, FRAT utilizes lightweight sampling subroutines to
construct a random strategy for the attacker. We prove that the continuous-time
limit of FRAT converges to a mixed Nash equilibria in a zero-sum game formed by
a defender and an attacker. Experimental results also demonstrate the
efficiency of FRAT on CIFAR-10 and CIFAR-100 datasets.Comment: Extended version of paper https://doi.org/10.1609/aaai.v37i9.26247
which appeared in AAAI 202