93 research outputs found
Imbalanced Gradients: A Subtle Cause of Overestimated Adversarial Robustness
Evaluating the robustness of a defense model is a challenging task in
adversarial robustness research. Obfuscated gradients, a type of gradient
masking, have previously been found to exist in many defense methods and cause
a false signal of robustness. In this paper, we identify a more subtle
situation called Imbalanced Gradients that can also cause overestimated
adversarial robustness. The phenomenon of imbalanced gradients occurs when the
gradient of one term of the margin loss dominates and pushes the attack towards
to a suboptimal direction. To exploit imbalanced gradients, we formulate a
Margin Decomposition (MD) attack that decomposes a margin loss into individual
terms and then explores the attackability of these terms separately via a
two-stage process. We also propose a MultiTargeted and an ensemble version of
our MD attack. By investigating 17 defense models proposed since 2018, we find
that 6 models are susceptible to imbalanced gradients and our MD attack can
decrease their robustness evaluated by the best baseline standalone attack by
another 2%. We also provide an in-depth analysis of the likely causes of
imbalanced gradients and effective countermeasures.Comment: 19 pages, 7 figue
Privacy and Robustness in Federated Learning: Attacks and Defenses
As data are increasingly being stored in different silos and societies
becoming more aware of data privacy issues, the traditional centralized
training of artificial intelligence (AI) models is facing efficiency and
privacy challenges. Recently, federated learning (FL) has emerged as an
alternative solution and continue to thrive in this new reality. Existing FL
protocol design has been shown to be vulnerable to adversaries within or
outside of the system, compromising data privacy and system robustness. Besides
training powerful global models, it is of paramount importance to design FL
systems that have privacy guarantees and are resistant to different types of
adversaries. In this paper, we conduct the first comprehensive survey on this
topic. Through a concise introduction to the concept of FL, and a unique
taxonomy covering: 1) threat models; 2) poisoning attacks and defenses against
robustness; 3) inference attacks and defenses against privacy, we provide an
accessible review of this important topic. We highlight the intuitions, key
techniques as well as fundamental assumptions adopted by various attacks and
defenses. Finally, we discuss promising future research directions towards
robust and privacy-preserving federated learning.Comment: arXiv admin note: text overlap with arXiv:2003.02133; text overlap
with arXiv:1911.11815 by other author
Reconstructive Neuron Pruning for Backdoor Defense
Deep neural networks (DNNs) have been found to be vulnerable to backdoor
attacks, raising security concerns about their deployment in mission-critical
applications. While existing defense methods have demonstrated promising
results, it is still not clear how to effectively remove backdoor-associated
neurons in backdoored DNNs. In this paper, we propose a novel defense called
\emph{Reconstructive Neuron Pruning} (RNP) to expose and prune backdoor neurons
via an unlearning and then recovering process. Specifically, RNP first unlearns
the neurons by maximizing the model's error on a small subset of clean samples
and then recovers the neurons by minimizing the model's error on the same data.
In RNP, unlearning is operated at the neuron level while recovering is operated
at the filter level, forming an asymmetric reconstructive learning procedure.
We show that such an asymmetric process on only a few clean samples can
effectively expose and prune the backdoor neurons implanted by a wide range of
attacks, achieving a new state-of-the-art defense performance. Moreover, the
unlearned model at the intermediate step of our RNP can be directly used to
improve other backdoor defense tasks including backdoor removal, trigger
recovery, backdoor label detection, and backdoor sample detection. Code is
available at \url{https://github.com/bboylyg/RNP}.Comment: Accepted by ICML2
On the Importance of Spatial Relations for Few-shot Action Recognition
Deep learning has achieved great success in video recognition, yet still
struggles to recognize novel actions when faced with only a few examples. To
tackle this challenge, few-shot action recognition methods have been proposed
to transfer knowledge from a source dataset to a novel target dataset with only
one or a few labeled videos. However, existing methods mainly focus on modeling
the temporal relations between the query and support videos while ignoring the
spatial relations. In this paper, we find that the spatial misalignment between
objects also occurs in videos, notably more common than the temporal
inconsistency. We are thus motivated to investigate the importance of spatial
relations and propose a more accurate few-shot action recognition method that
leverages both spatial and temporal information. Particularly, a novel Spatial
Alignment Cross Transformer (SA-CT) which learns to re-adjust the spatial
relations and incorporates the temporal information is contributed. Experiments
reveal that, even without using any temporal information, the performance of
SA-CT is comparable to temporal based methods on 3/4 benchmarks. To further
incorporate the temporal information, we propose a simple yet effective
Temporal Mixer module. The Temporal Mixer enhances the video representation and
improves the performance of the full SA-CT model, achieving very competitive
results. In this work, we also exploit large-scale pretrained models for
few-shot action recognition, providing useful insights for this research
direction
Fake Alignment: Are LLMs Really Aligned Well?
The growing awareness of safety concerns in large language models (LLMs) has
sparked considerable interest in the evaluation of safety within current
research endeavors. This study investigates an interesting issue pertaining to
the evaluation of LLMs, namely the substantial discrepancy in performance
between multiple-choice questions and open-ended questions. Inspired by
research on jailbreak attack patterns, we argue this is caused by mismatched
generalization. That is, the LLM does not have a comprehensive understanding of
the complex concept of safety. Instead, it only remembers what to answer for
open-ended safety questions, which makes it unable to solve other forms of
safety tests. We refer to this phenomenon as fake alignment and construct a
comparative benchmark to empirically verify its existence in LLMs. Such fake
alignment renders previous evaluation protocols unreliable. To address this, we
introduce the Fake alIgNment Evaluation (FINE) framework and two novel
metrics--Consistency Score (CS) and Consistent Safety Score (CSS), which
jointly assess two complementary forms of evaluation to quantify fake alignment
and obtain corrected performance estimates. Applying FINE to 14 widely-used
LLMs reveals several models with purported safety are poorly aligned in
practice. Our work highlights potential limitations in prevailing alignment
methodologies
Backdoor Attacks on Crowd Counting
Crowd counting is a regression task that estimates the number of people in a
scene image, which plays a vital role in a range of safety-critical
applications, such as video surveillance, traffic monitoring and flow control.
In this paper, we investigate the vulnerability of deep learning based crowd
counting models to backdoor attacks, a major security threat to deep learning.
A backdoor attack implants a backdoor trigger into a target model via data
poisoning so as to control the model's predictions at test time. Different from
image classification models on which most of existing backdoor attacks have
been developed and tested, crowd counting models are regression models that
output multi-dimensional density maps, thus requiring different techniques to
manipulate.
In this paper, we propose two novel Density Manipulation Backdoor Attacks
(DMBA and DMBA) to attack the model to produce arbitrarily large or
small density estimations. Experimental results demonstrate the effectiveness
of our DMBA attacks on five classic crowd counting models and four types of
datasets. We also provide an in-depth analysis of the unique challenges of
backdooring crowd counting models and reveal two key elements of effective
attacks: 1) full and dense triggers and 2) manipulation of the ground truth
counts or density maps. Our work could help evaluate the vulnerability of crowd
counting models to potential backdoor attacks.Comment: To appear in ACMMM 2022. 10pages, 6 figures and 2 table
Ultra-efficient frequency comb generation in AlGaAs-on-insulator microresonators
Recent advances in nonlinear optics have revolutionized integrated photonics, providing on-chip solutions to a wide range of new applications. Currently, state of the art integrated nonlinear photonic devices are mainly based on dielectric material platforms, such as Si₃N₄ and SiO₂. While semiconductor materials feature much higher nonlinear coefficients and convenience in active integration, they have suffered from high waveguide losses that prevent the realization of efficient nonlinear processes on-chip. Here, we challenge this status quo and demonstrate a low loss AlGaAs-on-insulator platform with anomalous dispersion and quality (Q) factors beyond 1.5 × 10⁶. Such a high quality factor, combined with high nonlinear coefficient and small mode volume, enabled us to demonstrate a Kerr frequency comb threshold of only ∼36 µW in a resonator with a 1 THz free spectral range, ∼100 times lower compared to that in previous semiconductor platforms. Moreover, combs with broad spans (>250 nm) have been generated with a pump power of ∼300 µW, which is lower than the threshold power of state-of the-art dielectric micro combs. A soliton-step transition has also been observed for the first time in an AlGaAs resonator
- …