241 research outputs found
On the minimax optimality and superiority of deep neural network learning over sparse parameter spaces
Deep learning has been applied to various tasks in the field of machine
learning and has shown superiority to other common procedures such as kernel
methods. To provide a better theoretical understanding of the reasons for its
success, we discuss the performance of deep learning and other methods on a
nonparametric regression problem with a Gaussian noise. Whereas existing
theoretical studies of deep learning have been based mainly on mathematical
theories of well-known function classes such as H\"{o}lder and Besov classes,
we focus on function classes with discontinuity and sparsity, which are those
naturally assumed in practice. To highlight the effectiveness of deep learning,
we compare deep learning with a class of linear estimators representative of a
class of shallow estimators. It is shown that the minimax risk of a linear
estimator on the convex hull of a target function class does not differ from
that of the original target function class. This results in the suboptimality
of linear methods over a simple but non-convex function class, on which deep
learning can attain nearly the minimax-optimal rate. In addition to this
extreme case, we consider function classes with sparse wavelet coefficients. On
these function classes, deep learning also attains the minimax rate up to log
factors of the sample size, and linear methods are still suboptimal if the
assumed sparsity is strong. We also point out that the parameter sharing of
deep neural networks can remarkably reduce the complexity of the model in our
setting.Comment: 33 page
Harmless Overparametrization in Two-layer Neural Networks
Overparametrized neural networks, where the number of active parameters is
larger than the sample size, prove remarkably effective in modern deep learning
practice. From the classical perspective, however, much fewer parameters are
sufficient for optimal estimation and prediction, whereas overparametrization
can be harmful even in the presence of explicit regularization. To reconcile
this conflict, we present a generalization theory for overparametrized ReLU
networks by incorporating an explicit regularizer based on the scaled variation
norm. Interestingly, this regularizer is equivalent to the ridge from the angle
of gradient-based optimization, but is similar to the group lasso in terms of
controlling model complexity. By exploiting this ridge-lasso duality, we show
that overparametrization is generally harmless to two-layer ReLU networks. In
particular, the overparametrized estimators are minimax optimal up to a
logarithmic factor. By contrast, we show that overparametrized random feature
models suffer from the curse of dimensionality and thus are suboptimal
Adaptive deep learning for nonlinear time series models
In this paper, we develop a general theory for adaptive nonparametric
estimation of the mean function of a non-stationary and nonlinear time series
model using deep neural networks (DNNs). We first consider two types of DNN
estimators, non-penalized and sparse-penalized DNN estimators, and establish
their generalization error bounds for general non-stationary time series. We
then derive minimax lower bounds for estimating mean functions belonging to a
wide class of nonlinear autoregressive (AR) models that include nonlinear
generalized additive AR, single index, and threshold AR models. Building upon
the results, we show that the sparse-penalized DNN estimator is adaptive and
attains the minimax optimal rates up to a poly-logarithmic factor for many
nonlinear AR models. Through numerical simulations, we demonstrate the
usefulness of the DNN methods for estimating nonlinear AR models with intrinsic
low-dimensional structures and discontinuous or rough mean functions, which is
consistent with our theory.Comment: 49 pages, 1 figur
- …