4 research outputs found
Lipschitz Networks and Distributional Robustness
Robust risk minimisation has several advantages: it has been studied with
regards to improving the generalisation properties of models and robustness to
adversarial perturbation. We bound the distributionally robust risk for a model
class rich enough to include deep neural networks by a regularised empirical
risk involving the Lipschitz constant of the model. This allows us to
interpretand quantify the robustness properties of a deep neural network. As an
application we show the distributionally robust risk upperbounds the
adversarial training risk
Nonparametric Online Learning Using Lipschitz Regularized Deep Neural Networks
Deep neural networks are considered to be state of the art models in many
offline machine learning tasks. However, their performance and generalization
abilities in online learning tasks are much less understood. Therefore, we
focus on online learning and tackle the challenging problem where the
underlying process is stationary and ergodic and thus removing the i.i.d.
assumption and allowing observations to depend on each other arbitrarily. We
prove the generalization abilities of Lipschitz regularized deep neural
networks and show that by using those networks, a convergence to the best
possible prediction strategy is guaranteed
Principal Component Analysis Based on T-norm Maximization
Classical principal component analysis (PCA) may suffer from the sensitivity
to outliers and noise. Therefore PCA based on -norm and -norm
() have been studied. Among them, the ones based on -norm
seem to be most interesting from the robustness point of view. However, their
numerical performance is not satisfactory. Note that, although T-norm
is similar to -norm () in some sense, it has the stronger
suppression effect to outliers and better continuity. So PCA based on
T-norm is proposed in this paper. Our numerical experiments have shown
that its performance is superior than PCA- and SPCA as well as
PCA, PCA- obviously
Monge blunts Bayes: Hardness Results for Adversarial Training
The last few years have seen a staggering number of empirical studies of the
robustness of neural networks in a model of adversarial perturbations of their
inputs. Most rely on an adversary which carries out local modifications within
prescribed balls. None however has so far questioned the broader picture: how
to frame a resource-bounded adversary so that it can be severely detrimental to
learning, a non-trivial problem which entails at a minimum the choice of loss
and classifiers.
We suggest a formal answer for losses that satisfy the minimal statistical
requirement of being proper. We pin down a simple sufficient property for any
given class of adversaries to be detrimental to learning, involving a central
measure of "harmfulness" which generalizes the well-known class of integral
probability metrics. A key feature of our result is that it holds for all
proper losses, and for a popular subset of these, the optimisation of this
central measure appears to be independent of the loss. When classifiers are
Lipschitz -- a now popular approach in adversarial training --, this
optimisation resorts to optimal transport to make a low-budget compression of
class marginals. Toy experiments reveal a finding recently separately observed:
training against a sufficiently budgeted adversary of this kind improves
generalization