306 research outputs found
PAC-Bayesian Spectrally-Normalized Bounds for Adversarially Robust Generalization
Deep neural networks (DNNs) are vulnerable to adversarial attacks. It is
found empirically that adversarially robust generalization is crucial in
establishing defense algorithms against adversarial attacks. Therefore, it is
interesting to study the theoretical guarantee of robust generalization. This
paper focuses on norm-based complexity, based on a PAC-Bayes approach
(Neyshabur et al., 2017). The main challenge lies in extending the key
ingredient, which is a weight perturbation bound in standard settings, to the
robust settings. Existing attempts heavily rely on additional strong
assumptions, leading to loose bounds. In this paper, we address this issue and
provide a spectrally-normalized robust generalization bound for DNNs. Compared
to existing bounds, our bound offers two significant advantages: Firstly, it
does not depend on additional assumptions. Secondly, it is considerably
tighter, aligning with the bounds of standard generalization. Therefore, our
result provides a different perspective on understanding robust generalization:
The mismatch terms between standard and robust generalization bounds shown in
previous studies do not contribute to the poor robust generalization. Instead,
these disparities solely due to mathematical issues. Finally, we extend the
main result to adversarial robustness against general non- attacks and
other neural network architectures.Comment: NeurIPS 202
On the sample complexity of adversarial multi-source PAC learning
We study the problem of learning from multiple untrusted data sources, a scenario of increasing practical relevance given the recent emergence of crowdsourcing and collaborative learning paradigms. Specifically, we analyze the situation in which a learning system obtains datasets from multiple sources, some of which might be biased or even adversarially perturbed. It is
known that in the single-source case, an adversary with the power to corrupt a fixed fraction of the training data can prevent PAC-learnability, that is, even in the limit of infinitely much training data, no learning system can approach the optimal test error. In this work we show that, surprisingly, the same is not true in the multi-source setting, where the adversary can arbitrarily
corrupt a fixed fraction of the data sources. Our main results are a generalization bound that provides finite-sample guarantees for this learning setting, as well as corresponding lower bounds. Besides establishing PAC-learnability our results also show that in a cooperative learning setting sharing data with other parties has provable benefits, even if some
participants are malicious
- β¦