28 research outputs found
Expressivity and Approximation Properties of Deep Neural Networks with ReLU Activation
In this paper, we investigate the expressivity and approximation properties
of deep neural networks employing the ReLU activation function for . Although deep ReLU networks can approximate polynomials effectively, deep
ReLU networks have the capability to represent higher-degree polynomials
precisely. Our initial contribution is a comprehensive, constructive proof for
polynomial representation using deep ReLU networks. This allows us to
establish an upper bound on both the size and count of network parameters.
Consequently, we are able to demonstrate a suboptimal approximation rate for
functions from Sobolev spaces as well as for analytic functions. Additionally,
through an exploration of the representation power of deep ReLU networks
for shallow networks, we reveal that deep ReLU networks can approximate
functions from a range of variation spaces, extending beyond those generated
solely by the ReLU activation function. This finding demonstrates the
adaptability of deep ReLU networks in approximating functions within
various variation spaces
ChebNet: Efficient and Stable Constructions of Deep Neural Networks with Rectified Power Units using Chebyshev Approximations
In a recent paper [B. Li, S. Tang and H. Yu, arXiv:1903.05858], it was shown
that deep neural networks built with rectified power units (RePU) can give
better approximation for sufficient smooth functions than those with rectified
linear units, by converting polynomial approximation given in power series into
deep neural networks with optimal complexity and no approximation error.
However, in practice, power series are not easy to compute. In this paper, we
propose a new and more stable way to construct deep RePU neural networks based
on Chebyshev polynomial approximations. By using a hierarchical structure of
Chebyshev polynomial approximation in frequency domain, we build efficient and
stable deep neural network constructions. In theory, ChebNets and the deep RePU
nets based on Power series have the same upper error bounds for general
function approximations. But numerically, ChebNets are much more stable.
Numerical results show that the constructed ChebNets can be further trained and
obtain much better results than those obtained by training deep RePU nets
constructed basing on power series.Comment: 18 pages, 6 figures, 2 table