The coupling effect of Lipschitz regularization in deep neural networks

We investigate robustness of deep feed-forward neural networks when input data are subject to random uncertainties. More specifically, we consider regularization of the network by its Lipschitz constant and emphasize its role. We highlight the fact that this regularization is not only a way to control the magnitude of the weights but has also a coupling effect on the network weights accross the layers. We claim and show evidence on a dataset that this coupling effect brings a tradeoff between robustness and expressiveness of the network. This suggests that Lipschitz regularization should be carefully implemented so as to maintain coupling accross layers

On the Use of Generative Adversarial Networks for Aircraft Trajectory Generation and Atypical Approach Detection

International audienceAircraft approach flight path safety management provides procedures that guide the aircraft to intercept the final approach axis and runway slope before landing. In order to detect atypical behavior, this paper explores the use of data generative models to learn real approach flight path probability distributions and identify flights that do not follow these distributions. Through the use of Generative Adversarial Networks (GAN), a GAN is first trained to learn real flight paths, generating new flights from learned distributions. Experiments show that the new generated flights follow realistic patterns. Unlike trajectories generated by physical models, the proposed technique, only based on past flight data, is able to account for external factors such as Air Traffic Control (ATC) orders, pilot behavior or meteorological phenomena. Next, the trained GAN is used to identify abnormal trajectories and compare the results with a clustering technique combined with a functional principal component analysis. The results show that reported non compliant trajectories are relevant

Primal-dual techniques for nonlinear programming and applications to artificial neural network training.

In this work, new developments in primal-dual techniques for general constrained non-linear programming problems are proposed. We first implement a modified version of the general nonlinear primal-dual algorithm that was published by El-Bakry et al. (21). We use the algorithm as a backbone of a new stochastic hybrid technique for solving general constrained nonlinear programming problems. The idea is to combine a fast local optimization strategy and a global search technique. The technique is a modified nonlinear primal-dual technique that uses concepts from simulated annealing to increase the probability of converging to the global minima of the objective function. At each iteration, the algorithm solves the Karush-Kuhn-Tucker optimality conditions to find the next iterate. A random noise is added to the resulting direction of move in order to escape local minima. The noise is gradually removed throughout the iteration process. We show that for complicated problems that possess numerous local minima and global minima, the proposed algorithm outperforms the deterministic approach. We also develop a new class of incremental nonlinear primal-dual techniques for solving optimization problems with special decomposition properties. Specifically, the objective functions of the problems are sums of independent nonconvex differentiable terms minimized subject to a set of nonlinear constraints for each term. The technique performs successive primal-dual increments for each decomposition term of the objective function. The method is particularly beneficial for online applications and problems that have a large amount of data. We show that the technique can be nicely applied to artificial neural training and provide experimental results for financial forecasting problems

The coupling effect of Lipschitz regularization in neural networks

International audienceWe investigate the robustness of feed-forward neural networks when input data are subject to random uncertainties. More specifically, we consider regularization of the network by its Lipschitz constant and emphasize its role. We highlight the fact that this regularization is not only a way to control the magnitude of the weights but has also a coupling effect on the network weights across the layers. We claim and show evidence on regression and classification datasets that this coupling effect brings a trade-of between robustness and expressiveness of the network. This suggests that Lipschitz regularization should be carefully implemented so as to maintain coupling across layers

A note on supervised classification and Nash-equilibrium problems

International audienceIn this note, we investigate connections between supervised classification and (Generalized) Nash equilibrium problems (NEP & GNEP). For the specific case of support vector machines (SVM), we exploit the geometric properties of class separation in the dual space to formulate a non-cooperative game. NEP and Generalized NEP formulations are proposed for both binary and multi-class SVM problems

Probabilistic Robustness Estimates for Deep Neural Networks

International audienceRobustness of deep neural networks is a critical issue in practical applications. In the case of deep dense neural networks, under random noise attacks, we propose to study the probability that the output of the network deviates from its nominal value by a given threshold. We derive a simple concentration inequality for the propagation of the input uncertainty through the network using the Cramer-Chernoff method and estimates of the local variation of the neural network mapping computed at the training points. We further discuss and exploit the resulting condition on the network to regularize the loss function during training. Finally, we assess the proposed tail probability estimate empirically on three public regression datasets and show that the observed robustness is very well estimated by the proposed method