Deep Stable neural networks: large-width asymptotics and convergence rates

In modern deep learning, there is a recent and growing literature on the interplay between large-width asymptotic properties of deep Gaussian neural networks (NNs), i.e. deep NNs with Gaussian-distributed weights, and Gaussian stochastic processes (SPs). Motivated by empirical analyses that show the potential of replacing Gaussian distributions with Stable distributions for the NN’s weights, in this paper we present a rigorous analysis of the large-width asymptotic behaviour of (fully connected) feed-forward deep Stable NNs, i.e. deep NNs with Stabledistributed weights. We show that as the width goes to infinity jointly over the NN’s layers, i.e. the “joint growth” setting, a rescaled deep Stable NN converges weakly to a Stable SP whose distribution is characterized recursively through the NN’s layers. Because of the non-triangular structure of the NN, this is a non-standard asymptotic problem, to which we propose an inductive approach of independent interest. Then, we establish sup-norm convergence rates of the rescaled deep Stable NN to the Stable SP, under the “joint growth” and a “sequential growth” of the width over the NN’s layers. Such a result provides the difference between the “joint growth” and the “sequential growth” settings, showing that the former leads to a slower rate than the latter, depending on the depth of the layer and the number of inputs of the NN. Our work extends some recent results on infinitely wide limits for deep Gaussian NNs to the more general deep Stable NNs, providing the first result on convergence rates in the “joint growth” setting

Stable behaviour of infinitely wide deep neural networks

We consider fully connected feed-forward deep neural networks (NNs) where weights and biases are independent and identically distributed as symmetric centered stable distributions. Then, we show that the infinite wide limit of the NN, under suitable scaling on the weights, is a stochastic process whose finite-dimensional distributions are multivariate stable distributions. The limiting process is referred to as the stable process, and it generalizes the class of Gaussian processes recently obtained as infinite wide limits of NNs (Matthews at al., 2018b). Parameters of the stable process can be computed via an explicit recursion over the layers of the network. Our result contributes to the theory of fully connected feed-forward deep NNs, and it paves the way to expand recent lines of research that rely on Gaussian infinite wide limits.Comment: 25 pages, 3 figure

Plasma Membranes in Psoriatic Cells. A Freeze-fracture Study

A freeze-fracture study of affected and unaffected psoriatic skin has demonstrated the presence of marked modification of the plasma membrane in the psoriatic lesion. In the lower layers of the epidermis, an increase of membrane associated particles was observed in many keratinocytes, possibly representing the morphological intramembranous equivalent of changes in the outer cell membrane demonstrated with cytochemical techniques. Furthermore, in the malphighian layer, numerous gap junctions have been found, which may be interpreted as a phenomenon compensating the uncontrolled proliferation, and may represent a point of differentiation between cell proliferation in psoriasis and neoplasia. This technique confirmed the poor tendency to adhesion of keratinocytes in extrajunctional areas, which had already been shown by other morphological techniques

Large-width functional asymptotics for deep Gaussian neural networks

In this paper, we consider fully connected feed-forward deep neural networks where weights and biases are independent and identically distributed according to Gaussian distributions. Extending previous results (Matthews et al., 2018a;b; Yang, 2019) we adopt a function-space perspective, i.e. we look at neural networks as infinite-dimensional random elements on the input space $\mathbb{R}^I$. Under suitable assumptions on the activation function we show that: i) a network defines a continuous Gaussian process on the input space $\mathbb{R}^I$; ii) a network with re-scaled weights converges weakly to a continuous Gaussian process in the large-width limit; iii) the limiting Gaussian process has almost surely locally $\gamma$-H\"older continuous paths, for $0 < \gamma <1$. Our results contribute to recent theoretical studies on the interplay between infinitely wide deep neural networks and Gaussian processes by establishing weak convergence in function-space with respect to a stronger metric

The strong weak convergence of the quasi-EA

In this paper, we investigate the convergence of a novel simulation scheme to the target diffusion process. This scheme, the Quasi-EA, is closely related to the Exact Algorithm (EA) for diffusion processes, as it is obtained by neglecting the rejection step in EA. We prove the existence of a myopic coupling between the Quasi-EA and the diffusion. Moreover, an upper bound for the coupling probability is given. Consequently we establish the convergence of the Quasi-EA to the diffusion with respect to the total variation distance

epsilon-Strong simulation of the Brownian path

We present an iterative sampling method which delivers upper and lower bounding processes for the Brownian path. We develop such processes with particular emphasis on being able to unbiasedly simulate them on a personal computer. The dominating processes converge almost surely in the supremum and L1L1 norms. In particular, the rate of converge in L1L1 is of the order O(K−1/2)O(K−1/2), KK denoting the computing cost. The a.s. enfolding of the Brownian path can be exploited in Monte Carlo applications involving Brownian paths whence our algorithm (termed the εε-strong algorithm) can deliver unbiased Monte Carlo estimators over path expectations, overcoming discretisation errors characterising standard approaches. We will show analytical results from applications of the εε-strong algorithm for estimating expectations arising in option pricing. We will also illustrate that individual steps of the algorithm can be of separate interest, giving new simulation methods for interesting Brownian distributions

An empirical study of the efficiency of EA for diffusion simulation

In this paper we investigate the efficiency of some simulation schemes for the numerical solution of a one dimensional stochastic differential equation (SDE). The schemes considered are: the Exact Algorithm (EA), the Euler, the Predictor-Corrector and the Ozaki- Shoji schemes. The focus of the work is on EA which samples skeletons of SDEs without any approximation. The analysis is carried out via a simulation study using some test SDEs. We also consider efficiency issues arising by the extension of EA to the multi-dimensional setting