10,273 research outputs found
Algorithms for Learning Sparse Additive Models with Interactions in High Dimensions
A function is a Sparse Additive
Model (SPAM), if it is of the form where , . Assuming 's, to be unknown, there exists extensive work
for estimating from its samples. In this work, we consider a generalized
version of SPAMs, that also allows for the presence of a sparse number of
second order interaction terms. For some , with , the function is now assumed to be of the form:
. Assuming we have the
freedom to query anywhere in its domain, we derive efficient algorithms
that provably recover with finite sample bounds.
Our analysis covers the noiseless setting where exact samples of are
obtained, and also extends to the noisy setting where the queries are corrupted
with noise. For the noisy setting in particular, we consider two noise models
namely: i.i.d Gaussian noise and arbitrary but bounded noise. Our main methods
for identification of essentially rely on estimation of sparse
Hessian matrices, for which we provide two novel compressed sensing based
schemes. Once are known, we show how the
individual components , can be estimated via
additional queries of , with uniform error bounds. Lastly, we provide
simulation results on synthetic data that validate our theoretical findings.Comment: To appear in Information and Inference: A Journal of the IMA. Made
following changes after review process: (a) Corrected typos throughout the
text. (b) Corrected choice of sampling distribution in Section 5, see eqs.
(5.2), (5.3). (c) More detailed comparison with existing work in Section 8.
(d) Added Section B in appendix on roots of cubic equatio
Sparse Additive Models
We present a new class of methods for high-dimensional nonparametric
regression and classification called sparse additive models (SpAM). Our methods
combine ideas from sparse linear modeling and additive nonparametric
regression. We derive an algorithm for fitting the models that is practical and
effective even when the number of covariates is larger than the sample size.
SpAM is closely related to the COSSO model of Lin and Zhang (2006), but
decouples smoothing and sparsity, enabling the use of arbitrary nonparametric
smoothers. An analysis of the theoretical properties of SpAM is given. We also
study a greedy estimator that is a nonparametric version of forward stepwise
regression. Empirical results on synthetic and real data are presented, showing
that SpAM can be effective in fitting sparse nonparametric models in high
dimensional data
- β¦