74 research outputs found
A Closer Look at the Adversarial Robustness of Deep Equilibrium Models
Deep equilibrium models (DEQs) refrain from the traditional layer-stacking
paradigm and turn to find the fixed point of a single layer. DEQs have achieved
promising performance on different applications with featured memory
efficiency. At the same time, the adversarial vulnerability of DEQs raises
concerns. Several works propose to certify robustness for monotone DEQs.
However, limited efforts are devoted to studying empirical robustness for
general DEQs. To this end, we observe that an adversarially trained DEQ
requires more forward steps to arrive at the equilibrium state, or even
violates its fixed-point structure. Besides, the forward and backward tracks of
DEQs are misaligned due to the black-box solvers. These facts cause gradient
obfuscation when applying the ready-made attacks to evaluate or adversarially
train DEQs. Given this, we develop approaches to estimate the intermediate
gradients of DEQs and integrate them into the attacking pipelines. Our
approaches facilitate fully white-box evaluations and lead to effective
adversarial defense for DEQs. Extensive experiments on CIFAR-10 validate the
adversarial robustness of DEQs competitive with deep networks of similar sizes.Comment: Accepted at NeurIPS 2022. Our code is available at
https://github.com/minicheshire/DEQ-White-Box-Robustnes
Defense against Adversarial Attacks Using High-Level Representation Guided Denoiser
Neural networks are vulnerable to adversarial examples, which poses a threat
to their application in security sensitive systems. We propose high-level
representation guided denoiser (HGD) as a defense for image classification.
Standard denoiser suffers from the error amplification effect, in which small
residual adversarial noise is progressively amplified and leads to wrong
classifications. HGD overcomes this problem by using a loss function defined as
the difference between the target model's outputs activated by the clean image
and denoised image. Compared with ensemble adversarial training which is the
state-of-the-art defending method on large images, HGD has three advantages.
First, with HGD as a defense, the target model is more robust to either
white-box or black-box adversarial attacks. Second, HGD can be trained on a
small subset of the images and generalizes well to other images and unseen
classes. Third, HGD can be transferred to defend models other than the one
guiding it. In NIPS competition on defense against adversarial attacks, our HGD
solution won the first place and outperformed other models by a large margin
Improving Adversarial Robustness of DEQs with Explicit Regulations Along the Neural Dynamics
Deep equilibrium (DEQ) models replace the multiple-layer stacking of
conventional deep networks with a fixed-point iteration of a single-layer
transformation. Having been demonstrated to be competitive in a variety of
real-world scenarios, the adversarial robustness of general DEQs becomes
increasingly crucial for their reliable deployment. Existing works improve the
robustness of general DEQ models with the widely-used adversarial training (AT)
framework, but they fail to exploit the structural uniquenesses of DEQ models.
To this end, we interpret DEQs through the lens of neural dynamics and find
that AT under-regulates intermediate states. Besides, the intermediate states
typically provide predictions with a high prediction entropy. Informed by the
correlation between the entropy of dynamical systems and their stability
properties, we propose reducing prediction entropy by progressively updating
inputs along the neural dynamics. During AT, we also utilize random
intermediate states to compute the loss function. Our methods regulate the
neural dynamics of DEQ models in this manner. Extensive experiments demonstrate
that our methods substantially increase the robustness of DEQ models and even
outperform the strong deep network baselines.Comment: Accepted at ICML 2023. Our code is available at
https://github.com/minicheshire/DEQ-Regulating-Neural-Dynamic
- …