Empirical Bounds on Linear Regions of Deep Rectifier Networks

We can compare the expressiveness of neural networks that use rectified linear units (ReLUs) by the number of linear regions, which reflect the number of pieces of the piecewise linear functions modeled by such networks. However, enumerating these regions is prohibitive and the known analytical bounds are identical for networks with same dimensions. In this work, we approximate the number of linear regions through empirical bounds based on features of the trained network and probabilistic inference. Our first contribution is a method to sample the activation patterns defined by ReLUs using universal hash functions. This method is based on a Mixed-Integer Linear Programming (MILP) formulation of the network and an algorithm for probabilistic lower bounds of MILP solution sets that we call MIPBound, which is considerably faster than exact counting and reaches values in similar orders of magnitude. Our second contribution is a tighter activation-based bound for the maximum number of linear regions, which is particularly stronger in networks with narrow layers. Combined, these bounds yield a fast proxy for the number of linear regions of a deep neural network.Comment: AAAI 202

Improving network generalization through selection of examples

In this work, we study how the selection of examples affects the learning procedure in a neural network and its relationship with the complexity of the function under study and its architecture. We focus on three different problems: parity, addition of two number and bitshifting implemented on feed-forward Neural Networks. For the parity problem, one of the most used problems for testing learning algorithms, we obtain the result that only the use of the whole set of examples assures global learnings. For the other two functions we show that generalization can be considerably improved with a particular selection of examples instead of a random one.Sistemas InteligentesRed de Universidades con Carreras en Informática (RedUNCI

Improving network generalization through selection of examples

In this work, we study how the selection of examples affects the learning procedure in a neural network and its relationship with the complexity of the function under study and its architecture. We focus on three different problems: parity, addition of two number and bitshifting implemented on feed-forward Neural Networks. For the parity problem, one of the most used problems for testing learning algorithms, we obtain the result that only the use of the whole set of examples assures global learnings. For the other two functions we show that generalization can be considerably improved with a particular selection of examples instead of a random one.Sistemas InteligentesRed de Universidades con Carreras en Informática (RedUNCI

Improving network generalization through selection of examples

In this work, we study how the selection of examples affects the learning procedure in a neural network and its relationship with the complexity of the function under study and its architecture. We focus on three different problems: parity, addition of two number and bitshifting implemented on feed-forward Neural Networks. For the parity problem, one of the most used problems for testing learning algorithms, we obtain the result that only the use of the whole set of examples assures global learnings. For the other two functions we show that generalization can be considerably improved with a particular selection of examples instead of a random one.Sistemas InteligentesRed de Universidades con Carreras en Informática (RedUNCI