2 research outputs found
Quasi-Equivalence of Width and Depth of Neural Networks
While classic studies proved that wide networks allow universal
approximation, recent research and successes of deep learning demonstrate the
power of the network depth. Based on a symmetric consideration, we investigate
if the design of artificial neural networks should have a directional
preference, and what the mechanism of interaction is between the width and
depth of a network. We address this fundamental question by establishing a
quasi-equivalence between the width and depth of ReLU networks. Specifically,
we formulate a transformation from an arbitrary ReLU network to a wide network
and a deep network for either regression or classification so that an
essentially same capability of the original network can be implemented. That
is, a deep regression/classification ReLU network has a wide equivalent, and
vice versa, subject to an arbitrarily small error. Interestingly, the
quasi-equivalence between wide and deep classification ReLU networks is a
data-driven version of the De Morgan law
Fundamental Limits of Deep Learning-Based Binary Classifiers Trained with Hinge Loss
Although deep learning (DL) has led to several breakthroughs in many
disciplines as diverse as chemistry, computer science, electrical engineering,
mathematics, medicine, neuroscience, and physics, a comprehensive understanding
of why and how DL is empirically successful remains fundamentally elusive. To
attack this fundamental problem and unravel the mysteries behind DL's empirical
successes, significant innovations toward a unified theory of DL have been
made. These innovations encompass nearly fundamental advances in optimization,
generalization, and approximation. Despite these advances, however, no work to
date has offered a way to quantify the testing performance of a DL-based
algorithm employed to solve a pattern classification problem. To overcome this
fundamental challenge in part, this paper exposes the fundamental testing
performance limits of DL-based binary classifiers trained with hinge loss. For
binary classifiers that are based on deep rectified linear unit (ReLU)
feedforward neural networks (FNNs) and ones that are based on deep FNNs with
ReLU and Tanh activation, we derive their respective novel asymptotic testing
performance limits. The derived testing performance limits are validated by
extensive computer experiments