502 research outputs found
Statistical inference in compound functional models
We consider a general nonparametric regression model called the compound
model. It includes, as special cases, sparse additive regression and
nonparametric (or linear) regression with many covariates but possibly a small
number of relevant covariates. The compound model is characterized by three
main parameters: the structure parameter describing the "macroscopic" form of
the compound function, the "microscopic" sparsity parameter indicating the
maximal number of relevant covariates in each component and the usual
smoothness parameter corresponding to the complexity of the members of the
compound. We find non-asymptotic minimax rate of convergence of estimators in
such a model as a function of these three parameters. We also show that this
rate can be attained in an adaptive way
Nonparametric regression using deep neural networks with ReLU activation function
Consider the multivariate nonparametric regression model. It is shown that
estimators based on sparsely connected deep neural networks with ReLU
activation function and properly chosen network architecture achieve the
minimax rates of convergence (up to -factors) under a general
composition assumption on the regression function. The framework includes many
well-studied structural constraints such as (generalized) additive models.
While there is a lot of flexibility in the network architecture, the tuning
parameter is the sparsity of the network. Specifically, we consider large
networks with number of potential network parameters exceeding the sample size.
The analysis gives some insights into why multilayer feedforward neural
networks perform well in practice. Interestingly, for ReLU activation function
the depth (number of layers) of the neural network architectures plays an
important role and our theory suggests that for nonparametric regression,
scaling the network depth with the sample size is natural. It is also shown
that under the composition assumption wavelet estimators can only achieve
suboptimal rates.Comment: article, rejoinder and supplementary materia
- …