A stochastic Remes algorithm

AbstractWe treat the linear Tchebycheff approximation problem for a regression function ƒ ϵ C2[0, 1]. A stochastic Remes algorithm which uses only estimates of first derivatives is proposed and investigated for almost sure convergence and rate of convergence

Strongly consistent density estimation of regression residual

Consider the regression problem with a response variable Y and with a d-dimensional feature vector X. For the regression function m(x) = E {Y|X} = xg, this paper investigates methods for estimating the density of the residual Y - m(X) from independent and identically distributed data. For heteroscedastic regression, we prove the strong universal (density-free) L1-consistency of a recursive and a nonrecursive kernel density estimate based on a regression estimate

Detecting ineffective features for pattern recognition

For a binary classification problem, the hypothesis testing is studied, that a component of the observation vector is not effective, i.e., that component carries no information for the classification. We introduce nearest neighbor and partitioning estimates of the Bayes error probability, which result in a strongly consistent test

Rate of convergence of the density estimation of regression residual

Consider the regression problem with a response variable Y and with a d-dimensional feature vector X. For the regression function m(x) = E{Y|X = x}, this paper investigates methods for estimating the density of the residual Y - m(X) from independent and identically distributed data. If the density is twice differentiable and has compact support then we bound the rate of convergence of the kernel density estimate. It turns out that for d <_ 3 and for partitioning regression estimates, the regression estimation error has no influence in the rate of convergence of the density estimate

Exact rate of convergence of k-nearest-neighbor classification rule

A binary classification problem is considered. The excess error probability of the k-nearest neighbor classification rule according to the error probability of the Bayes decision is revisited by a decomposition of the excess error probability into approximation and estimation error. Under a weak margin condition and under a modified Lipschitz condition, tight upper bounds are presented such that one avoids the condition that the feature vector is bounded

Lossless Transformations and Excess Risk Bounds in Statistical Inference

We study the excess minimum risk in statistical inference, defined as the difference between the minimum expected loss in estimating a random variable from an observed feature vector and the minimum expected loss in estimating the same random variable from a transformation (statistic) of the feature vector. After characterizing lossless transformations, i.e., transformations for which the excess risk is zero for all loss functions, we construct a partitioning test statistic for the hypothesis that a given transformation is lossless and show that for i.i.d. data the test is strongly consistent. More generally, we develop information-theoretic upper bounds on the excess risk that uniformly hold over fairly general classes of loss functions. Based on these bounds, we introduce the notion of a delta-lossless transformation and give sufficient conditions for a given transformation to be universally delta-lossless. Applications to classification, nonparametric regression, portfolio strategies, information bottleneck, and deep learning, are also surveyed.Comment: to appear in Entrop