156 research outputs found
Principles of Neural Network Architecture Design - Invertibility and Domain Knowledge
Neural networks architectures allow a tremendous variety of design choices. In this work, we study two principles underlying these architectures: First, the design and application of invertible neural networks (INNs). Second, the incorporation of domain knowledge into neural network architectures. After introducing the mathematical foundations of deep learning, we address the invertibility of standard feedforward neural networks from a mathematical perspective. These results serve as a motivation for our proposed invertible residual networks (i-ResNets). This architecture class is then studied in two scenarios: First, we propose ways to use i-ResNets as a normalizing flow and demonstrate the applicability for high-dimensional generative modeling. Second, we study the excessive invariance of common deep image classifiers and discuss consequences for adversarial robustness. We finish with a study of convolutional neural networks for tumor classification based on imaging mass spectrometry (IMS) data. For this application, we propose an adapted architecture guided by our knowledge of the domain of IMS data and show its superior performance on two challenging tumor classification datasets
Signal Recovery from Pooling Representations
In this work we compute lower Lipschitz bounds of pooling operators
for as well as pooling operators preceded by
half-rectification layers. These give sufficient conditions for the design of
invertible neural network layers. Numerical experiments on MNIST and image
patches confirm that pooling layers can be inverted with phase recovery
algorithms. Moreover, the regularity of the inverse pooling, controlled by the
lower Lipschitz constant, is empirically verified with a nearest neighbor
regression.Comment: 17 pages, 3 figure
Residual Networks as Flows of Diffeomorphisms
International audienceThis paper addresses the understanding and characterization of residual networks (ResNet), which are among the state-of-the-art deep learning architectures for a variety of supervised learning problems. We focus on the mapping component of ResNets, which map the embedding space towards a new unknown space where the prediction or classification can be stated according to linear criteria. We show that this mapping component can be regarded as the numerical implementation of continuous flows of diffeomorphisms governed by ordinary differential equations. Especially, ResNets with shared weights are fully characterized as numerical approximation of exponential diffeomorphic operators. We stress both theoretically and numerically the relevance of the enforcement of diffeormorphic properties and the importance of numerical issues to make consistent the continuous formulation and the discretized ResNet implementation. We further discuss the resulting theoretical and computational insights on ResNet architectures
- âŠ