15,771 research outputs found
Direct Estimation of Information Divergence Using Nearest Neighbor Ratios
We propose a direct estimation method for R\'{e}nyi and f-divergence measures
based on a new graph theoretical interpretation. Suppose that we are given two
sample sets and , respectively with and samples, where
is a constant value. Considering the -nearest neighbor (-NN)
graph of in the joint data set , we show that the average powered
ratio of the number of points to the number of points among all -NN
points is proportional to R\'{e}nyi divergence of and densities. A
similar method can also be used to estimate f-divergence measures. We derive
bias and variance rates, and show that for the class of -H\"{o}lder
smooth functions, the estimator achieves the MSE rate of
. Furthermore, by using a weighted ensemble
estimation technique, for density functions with continuous and bounded
derivatives of up to the order , and some extra conditions at the support
set boundary, we derive an ensemble estimator that achieves the parametric MSE
rate of . Our estimators are more computationally tractable than other
competing estimators, which makes them appealing in many practical
applications.Comment: 2017 IEEE International Symposium on Information Theory (ISIT
Sparse Model Identification and Learning for Ultra-high-dimensional Additive Partially Linear Models
The additive partially linear model (APLM) combines the flexibility of
nonparametric regression with the parsimony of regression models, and has been
widely used as a popular tool in multivariate nonparametric regression to
alleviate the "curse of dimensionality". A natural question raised in practice
is the choice of structure in the nonparametric part, that is, whether the
continuous covariates enter into the model in linear or nonparametric form. In
this paper, we present a comprehensive framework for simultaneous sparse model
identification and learning for ultra-high-dimensional APLMs where both the
linear and nonparametric components are possibly larger than the sample size.
We propose a fast and efficient two-stage procedure. In the first stage, we
decompose the nonparametric functions into a linear part and a nonlinear part.
The nonlinear functions are approximated by constant spline bases, and a triple
penalization procedure is proposed to select nonzero components using adaptive
group LASSO. In the second stage, we refit data with selected covariates using
higher order polynomial splines, and apply spline-backfitted local-linear
smoothing to obtain asymptotic normality for the estimators. The procedure is
shown to be consistent for model structure identification. It can identify
zero, linear, and nonlinear components correctly and efficiently. Inference can
be made on both linear coefficients and nonparametric functions. We conduct
simulation studies to evaluate the performance of the method and apply the
proposed method to a dataset on the Shoot Apical Meristem (SAM) of maize
genotypes for illustration
- …