188 research outputs found
Fast robust correlation for high-dimensional data
The product moment covariance is a cornerstone of multivariate data analysis,
from which one can derive correlations, principal components, Mahalanobis
distances and many other results. Unfortunately the product moment covariance
and the corresponding Pearson correlation are very susceptible to outliers
(anomalies) in the data. Several robust measures of covariance have been
developed, but few are suitable for the ultrahigh dimensional data that are
becoming more prevalent nowadays. For that one needs methods whose computation
scales well with the dimension, are guaranteed to yield a positive semidefinite
covariance matrix, and are sufficiently robust to outliers as well as
sufficiently accurate in the statistical sense of low variability. We construct
such methods using data transformations. The resulting approach is simple, fast
and widely applicable. We study its robustness by deriving influence functions
and breakdown values, and computing the mean squared error on contaminated
data. Using these results we select a method that performs well overall. This
also allows us to construct a faster version of the DetectDeviatingCells method
(Rousseeuw and Van den Bossche, 2018) to detect cellwise outliers, that can
deal with much higher dimensions. The approach is illustrated on genomic data
with 12,000 variables and color video data with 920,000 dimensions
Discussion of "The power of monitoring"
This is an invited comment on the discussion paper "The power of monitoring:
how to make the most of a contaminated multivariate sample" by A. Cerioli, M.
Riani, A. Atkinson and A. Corbellini that will appear in the journal
Statistical Methods & Applications
Finding Outliers in Surface Data and Video
Surface, image and video data can be considered as functional data with a
bivariate domain. To detect outlying surfaces or images, a new method is
proposed based on the mean and the variability of the degree of outlyingness at
each grid point. A rule is constructed to flag the outliers in the resulting
functional outlier map. Heatmaps of their outlyingness indicate the regions
which are most deviating from the regular surfaces. The method is applied to
fluorescence excitation-emission spectra after fitting a PARAFAC model, to MRI
image data which are augmented with their gradients, and to video surveillance
data
A generalized spatial sign covariance matrix
The well-known spatial sign covariance matrix (SSCM) carries out a radial
transform which moves all data points to a sphere, followed by computing the
classical covariance matrix of the transformed data. Its popularity stems from
its robustness to outliers, fast computation, and applications to correlation
and principal component analysis. In this paper we study more general radial
functions. It is shown that the eigenvectors of the generalized SSCM are still
consistent and the ranks of the eigenvalues are preserved. The influence
function of the resulting scatter matrix is derived, and it is shown that its
breakdown value is as high as that of the original SSCM. A simulation study
indicates that the best results are obtained when the inner half of the data
points are not transformed and points lying far away are moved to the center
- …