34 research outputs found
On Lipschitz Regularization of Convolutional Layers using Toeplitz Matrix Theory
This paper tackles the problem of Lipschitz regularization of Convolutional
Neural Networks. Lipschitz regularity is now established as a key property of
modern deep learning with implications in training stability, generalization,
robustness against adversarial examples, etc. However, computing the exact
value of the Lipschitz constant of a neural network is known to be NP-hard.
Recent attempts from the literature introduce upper bounds to approximate this
constant that are either efficient but loose or accurate but computationally
expensive. In this work, by leveraging the theory of Toeplitz matrices, we
introduce a new upper bound for convolutional layers that is both tight and
easy to compute. Based on this result we devise an algorithm to train Lipschitz
regularized Convolutional Neural Networks
Robust Neural Networks using Randomized Adversarial Training
Since the discovery of adversarial examples in machine learning, researchers have designed several techniques to train neural networks that are robust against different types of attacks (most notably ∞ and 2 based attacks). However , it has been observed that the defense mechanisms designed to protect against one type of attack often offer poor performance against the other. In this paper, we introduce Randomized Adversarial Training (RAT), a technique that is efficient both against 2 and ∞ attacks. To obtain this result, we build upon adversarial training, a technique that is efficient against ∞ attacks, and demonstrate that adding random noise at training and inference time further improves performance against 2 attacks. We then show that RAT is as efficient as adversarial training against ∞ attacks while being robust against strong 2 attacks. Our final comparative experiments demonstrate that RAT outperforms all state-of-the-art approaches against 2 and ∞ attacks
Precision-Recall Divergence Optimization for Generative Modeling with GANs and Normalizing Flows
Achieving a balance between image quality (precision) and diversity (recall)
is a significant challenge in the domain of generative models. Current
state-of-the-art models primarily rely on optimizing heuristics, such as the
Fr\'echet Inception Distance. While recent developments have introduced
principled methods for evaluating precision and recall, they have yet to be
successfully integrated into the training of generative models. Our main
contribution is a novel training method for generative models, such as
Generative Adversarial Networks and Normalizing Flows, which explicitly
optimizes a user-defined trade-off between precision and recall. More
precisely, we show that achieving a specified precision-recall trade-off
corresponds to minimizing a unique -divergence from a family we call the
\mbox{\em PR-divergences}. Conversely, any -divergence can be written as a
linear combination of PR-divergences and corresponds to a weighted
precision-recall trade-off. Through comprehensive evaluations, we show that our
approach improves the performance of existing state-of-the-art models like
BigGAN in terms of either precision or recall when tested on datasets such as
ImageNet
Randomization for adversarial robustness: the Good, the Bad and the Ugly
Deep neural networks are known to be vulnerable to adversarial attacks: A
small perturbation that is imperceptible to a human can easily make a
well-trained deep neural network misclassify. To defend against adversarial
attacks, randomized classifiers have been proposed as a robust alternative to
deterministic ones. In this work we show that in the binary classification
setting, for any randomized classifier, there is always a deterministic
classifier with better adversarial risk. In other words, randomization is not
necessary for robustness. In many common randomization schemes, the
deterministic classifiers with better risk are explicitly described: For
example, we show that ensembles of classifiers are more robust than mixtures of
classifiers, and randomized smoothing is more robust than input noise
injection. Finally, experiments confirm our theoretical results with the two
families of randomized classifiers we analyze.Comment: 8 pages + bibliography and appendix, 3 figures. Submitted to ICML
202
Un algorithme de fouille de données générique et parallèle pour architecture multi-coeurs
In the pattern mining field, there exist a large number of algorithms that can solve a large variety of distinct but similar pattern mining problems. This variety prevent broad adoption of data analysis with pattern mining algorithms. In this thesis we propose a formal framework that is able to capture a broad range of pattern mining problems. We illustrate the generality of our framework by formalizing three different pattern mining problems: the problem of closed frequent itemset mining, the problem of closed relational graph mining and the problem of closed gradual itemset mining. Building on this framework, we have designed ParaMiner, a generic and parallel algorithm for pattern mining. ParaMiner is able to solve any pattern mining problem that can be formalized within our framework. In order to achieve practical efficiency we have generalized important optimizations from state of the art algorithms and we have made ParaMiner able to exploit parallel computing platforms. We have conducted thorough experiments that demonstrate that despite being a generic algorithm, ParaMiner can compete with the fastest ad-hoc algorithms.Dans le domaine de l'extraction de motifs, il existe un grand nombre d'algorithmes pour résoudre une large variété de sous problèmes sensiblement identiques. Cette variété d'algorithmes freine l'adoption des techniques d'extraction de motifs pour l'analyse de données. Dans cette thèse, nous proposons un formalisme qui permet de capturer une large gamme de problèmes d'extraction de motifs. Pour démontrer la généralité de ce formalisme, nous l'utilisons pour décrire trois problèmes d'extraction de motifs : le problème d'extraction d'itemsets fréquents fermés, le problème d'extraction de graphes relationnels fermés ou le problème d'extraction d'itemsets graduels fermés. Ce formalisme nous permet de construire ParaMiner qui est un algorithme générique et parallèle pour les problèmes d'extraction de motifs. ParaMiner est capable de résoudre tous les problèmes d'extraction de motifs qui peuvent ˆtre décrit dans notre formalisme. Pour obtenir de bonne performances, nous avons généralisé plusieurs optimisations proposées par la communauté dans le cadre de problèmes spécifique d'extraction de motifs. Nous avons également exploité la puissance de calcul parallèle disponible dans les archi- tectures parallèles. Nos expériences démontrent qu'en dépit de la généricité de ParaMiner ses performances sont comparables avec celles obtenues par les algorithmes les plus rapides de l'état de l'art. Ces algorithmes bénéficient pourtant d'un avantage important, puisqu'ils incorporent de nombreuses optimisations spécifiques au sous problème d'extraction de motifs qu'ils résolvent
A generic and parallel pattern mining algorithm for multi-core architectures.
Dans le domaine de l'extraction de motifs, il existe un grand nombre d'algorithmes pour résoudre une large variété de sous problèmes sensiblement identiques. Cette variété d'algorithmes freine l'adoption des techniques d'extraction de motifs pour l'analyse de données. Dans cette thèse, nous proposons un formalisme qui permet de capturer une large gamme de problèmes d'extraction de motifs. Pour démontrer la généralité de ce formalisme, nous l'utilisons pour décrire trois problèmes d'extraction de motifs : le problème d'extraction d'itemsets fréquents fermés, le problème d'extraction de graphes relationnels fermés ou le problème d'extraction d'itemsets graduels fermés. Ce formalisme nous permet de construire ParaMiner qui est un algorithme générique et parallèle pour les problèmes d'extraction de motifs. ParaMiner est capable de résoudre tous les problèmes d'extraction de motifs qui peuvent ˆtre décrit dans notre formalisme. Pour obtenir de bonne performances, nous avons généralisé plusieurs optimisations proposées par la communauté dans le cadre de problèmes spécifique d'extraction de motifs. Nous avons également exploité la puissance de calcul parallèle disponible dans les archi- tectures parallèles. Nos expériences démontrent qu'en dépit de la généricité de ParaMiner ses performances sont comparables avec celles obtenues par les algorithmes les plus rapides de l'état de l'art. Ces algorithmes bénéficient pourtant d'un avantage important, puisqu'ils incorporent de nombreuses optimisations spécifiques au sous problème d'extraction de motifs qu'ils résolvent.In the pattern mining field, there exist a large number of algorithms that can solve a large variety of distinct but similar pattern mining problems. This variety prevent broad adoption of data analysis with pattern mining algorithms. In this thesis we propose a formal framework that is able to capture a broad range of pattern mining problems. We illustrate the generality of our framework by formalizing three different pattern mining problems: the problem of closed frequent itemset mining, the problem of closed relational graph mining and the problem of closed gradual itemset mining. Building on this framework, we have designed ParaMiner, a generic and parallel algorithm for pattern mining. ParaMiner is able to solve any pattern mining problem that can be formalized within our framework. In order to achieve practical efficiency we have generalized important optimizations from state of the art algorithms and we have made ParaMiner able to exploit parallel computing platforms. We have conducted thorough experiments that demonstrate that despite being a generic algorithm, ParaMiner can compete with the fastest ad-hoc algorithms
Constraint-based sequence mining using constraint programming
The goal of constraint-based sequence mining is to find sequences
of symbols that are included in a large number of input sequences
and that satisfy some constraints specified by the user. Many
constraints have been proposed in the literature, but a general
framework is still missing. We investigate the use of constraint
programming as general framework for this task. We first
identify four categories of constraints that are applicable to
sequence mining. We then propose two constraint programming
formulations. The first formulation introduces a new global
constraint called exists-embedding.
This formulation is the most efficient but does not support one
type of constraint. To support such constraints, we develop a
second formulation that is more general but incurs more
overhead. Both formulations can use the projected database
technique used in specialised algorithms. Experiments
demonstrate the flexibility towards constraint-based settings and
compare the approach to existing methods.In Integration of AI and OR Techniques in Constraint Programming
(CPAIOR), 2015status: publishe