190 research outputs found
Fast robust correlation for high-dimensional data
The product moment covariance is a cornerstone of multivariate data analysis,
from which one can derive correlations, principal components, Mahalanobis
distances and many other results. Unfortunately the product moment covariance
and the corresponding Pearson correlation are very susceptible to outliers
(anomalies) in the data. Several robust measures of covariance have been
developed, but few are suitable for the ultrahigh dimensional data that are
becoming more prevalent nowadays. For that one needs methods whose computation
scales well with the dimension, are guaranteed to yield a positive semidefinite
covariance matrix, and are sufficiently robust to outliers as well as
sufficiently accurate in the statistical sense of low variability. We construct
such methods using data transformations. The resulting approach is simple, fast
and widely applicable. We study its robustness by deriving influence functions
and breakdown values, and computing the mean squared error on contaminated
data. Using these results we select a method that performs well overall. This
also allows us to construct a faster version of the DetectDeviatingCells method
(Rousseeuw and Van den Bossche, 2018) to detect cellwise outliers, that can
deal with much higher dimensions. The approach is illustrated on genomic data
with 12,000 variables and color video data with 920,000 dimensions
Discussion of "The power of monitoring"
This is an invited comment on the discussion paper "The power of monitoring:
how to make the most of a contaminated multivariate sample" by A. Cerioli, M.
Riani, A. Atkinson and A. Corbellini that will appear in the journal
Statistical Methods & Applications
Finding Outliers in Surface Data and Video
Surface, image and video data can be considered as functional data with a
bivariate domain. To detect outlying surfaces or images, a new method is
proposed based on the mean and the variability of the degree of outlyingness at
each grid point. A rule is constructed to flag the outliers in the resulting
functional outlier map. Heatmaps of their outlyingness indicate the regions
which are most deviating from the regular surfaces. The method is applied to
fluorescence excitation-emission spectra after fitting a PARAFAC model, to MRI
image data which are augmented with their gradients, and to video surveillance
data
Measuring overlap in logistic regression
In this paper we show that the recent notion of regression depth can be used as a data-analytic tool to measure the amount of separation between successes and failures in the binary response framework. Extending this algorithm allows us to compute the overlap in data sets which are commonly fitted by logistic regression models. The overlap is the number of observations that would need to be removed to obtain complete or quasicomplete separation, i.e. the situation where the logistic regression parameters are no longer identifiable and the maximum likelihood estimate does not exist. It turns out that the overlap is often quite small
- …