287,387 research outputs found
Robust adaptive variable selection in ultra-high dimensional linear regression models
We consider the problem of simultaneous variable selection and estimation of
the corresponding regression coefficients in an ultra-high dimensional linear
regression models, an extremely important problem in the recent era. The
adaptive penalty functions are used in this regard to achieve the oracle
variable selection property along with easier computational burden. However,
the usual adaptive procedures (e.g., adaptive LASSO) based on the squared error
loss function is extremely non-robust in the presence of data contamination
which are quite common with large-scale data (e.g., noisy gene expression data,
spectra and spectral data). In this paper, we present a regularization
procedure for the ultra-high dimensional data using a robust loss function
based on the popular density power divergence (DPD) measure along with the
adaptive LASSO penalty. We theoretically study the robustness and the
large-sample properties of the proposed adaptive robust estimators for a
general class of error distributions; in particular, we show that the proposed
adaptive DPD-LASSO estimator is highly robust, satisfies the oracle variable
selection property, and the corresponding estimators of the regression
coefficients are consistent and asymptotically normal under easily verifiable
set of assumptions. Numerical illustrations are provided for the mostly used
normal error density. Finally, the proposal is applied to analyze an
interesting spectral dataset, in the field of chemometrics, regarding the
electron-probe X-ray microanalysis (EPXMA) of archaeological glass vessels from
the 16th and 17th centuries.Comment: Pre-print, Under revie
Renyi Differential Privacy of Propose-Test-Release and Applications to Private and Robust Machine Learning
Propose-Test-Release (PTR) is a differential privacy framework that works
with local sensitivity of functions, instead of their global sensitivity. This
framework is typically used for releasing robust statistics such as median or
trimmed mean in a differentially private manner. While PTR is a common
framework introduced over a decade ago, using it in applications such as robust
SGD where we need many adaptive robust queries is challenging. This is mainly
due to the lack of Renyi Differential Privacy (RDP) analysis, an essential
ingredient underlying the moments accountant approach for differentially
private deep learning. In this work, we generalize the standard PTR and derive
the first RDP bound for it when the target function has bounded global
sensitivity. We show that our RDP bound for PTR yields tighter DP guarantees
than the directly analyzed (\eps, \delta)-DP. We also derive the
algorithm-specific privacy amplification bound of PTR under subsampling. We
show that our bound is much tighter than the general upper bound and close to
the lower bound. Our RDP bounds enable tighter privacy loss calculation for the
composition of many adaptive runs of PTR. As an application of our analysis, we
show that PTR and our theoretical results can be used to design differentially
private variants for byzantine robust training algorithms that use robust
statistics for gradients aggregation. We conduct experiments on the settings of
label, feature, and gradient corruption across different datasets and
architectures. We show that PTR-based private and robust training algorithm
significantly improves the utility compared with the baseline.Comment: NeurIPS 202
Asymptotic equivalence and adaptive estimation for robust nonparametric regression
Asymptotic equivalence theory developed in the literature so far are only for
bounded loss functions. This limits the potential applications of the theory
because many commonly used loss functions in statistical inference are
unbounded. In this paper we develop asymptotic equivalence results for robust
nonparametric regression with unbounded loss functions. The results imply that
all the Gaussian nonparametric regression procedures can be robustified in a
unified way. A key step in our equivalence argument is to bin the data and then
take the median of each bin. The asymptotic equivalence results have
significant practical implications. To illustrate the general principles of the
equivalence argument we consider two important nonparametric inference
problems: robust estimation of the regression function and the estimation of a
quadratic functional. In both cases easily implementable procedures are
constructed and are shown to enjoy simultaneously a high degree of robustness
and adaptivity. Other problems such as construction of confidence sets and
nonparametric hypothesis testing can be handled in a similar fashion.Comment: Published in at http://dx.doi.org/10.1214/08-AOS681 the Annals of
Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical
Statistics (http://www.imstat.org
Robust methods for inferring sparse network structures
This is the post-print version of the final paper published in Computational Statistics & Data Analysis. The published article is available from the link below. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. Copyright @ 2013 Elsevier B.V.Networks appear in many fields, from finance to medicine, engineering, biology and social science. They often comprise of a very large number of entities, the nodes, and the interest lies in inferring the interactions between these entities, the edges, from relatively limited data. If the underlying network of interactions is sparse, two main statistical approaches are used to retrieve such a structure: covariance modeling approaches with a penalty constraint that encourages sparsity of the network, and nodewise regression approaches with sparse regression methods applied at each node. In the presence of outliers or departures from normality, robust approaches have been developed which relax the assumption of normality. Robust covariance modeling approaches are reviewed and compared with novel nodewise approaches where robust methods are used at each node. For low-dimensional problems, classical deviance tests are also included and compared with penalized likelihood approaches. Overall, copula approaches are found to perform best: they are comparable to the other methods under an assumption of normality or mild departures from this, but they are superior to the other methods when the assumption of normality is strongly violated
- …