625 research outputs found
Iteratively Reweighted Least Squares Algorithms for L1-Norm Principal Component Analysis
Principal component analysis (PCA) is often used to reduce the dimension of
data by selecting a few orthonormal vectors that explain most of the variance
structure of the data. L1 PCA uses the L1 norm to measure error, whereas the
conventional PCA uses the L2 norm. For the L1 PCA problem minimizing the
fitting error of the reconstructed data, we propose an exact reweighted and an
approximate algorithm based on iteratively reweighted least squares. We provide
convergence analyses, and compare their performance against benchmark
algorithms in the literature. The computational experiment shows that the
proposed algorithms consistently perform best
Distributed estimation from relative measurements of heterogeneous and uncertain quality
This paper studies the problem of estimation from relative measurements in a
graph, in which a vector indexed over the nodes has to be reconstructed from
pairwise measurements of differences between its components associated to nodes
connected by an edge. In order to model heterogeneity and uncertainty of the
measurements, we assume them to be affected by additive noise distributed
according to a Gaussian mixture. In this original setup, we formulate the
problem of computing the Maximum-Likelihood (ML) estimates and we design two
novel algorithms, based on Least Squares regression and
Expectation-Maximization (EM). The first algorithm (LS- EM) is centralized and
performs the estimation from relative measurements, the soft classification of
the measurements, and the estimation of the noise parameters. The second
algorithm (Distributed LS-EM) is distributed and performs estimation and soft
classification of the measurements, but requires the knowledge of the noise
parameters. We provide rigorous proofs of convergence of both algorithms and we
present numerical experiments to evaluate and compare their performance with
classical solutions. The experiments show the robustness of the proposed
methods against different kinds of noise and, for the Distributed LS-EM,
against errors in the knowledge of noise parameters.Comment: Submitted to IEEE transaction
All-In-One Robust Estimator of the Gaussian Mean
The goal of this paper is to show that a single robust estimator of the mean
of a multivariate Gaussian distribution can enjoy five desirable properties.
First, it is computationally tractable in the sense that it can be computed in
a time which is at most polynomial in dimension, sample size and the logarithm
of the inverse of the contamination rate. Second, it is equivariant by
translations, uniform scaling and orthogonal transformations. Third, it has a
high breakdown point equal to , and a nearly-minimax-rate-breakdown point
approximately equal to . Fourth, it is minimax rate optimal, up to a
logarithmic factor, when data consists of independent observations corrupted by
adversarially chosen outliers. Fifth, it is asymptotically efficient when the
rate of contamination tends to zero. The estimator is obtained by an iterative
reweighting approach. Each sample point is assigned a weight that is
iteratively updated by solving a convex optimization problem. We also establish
a dimension-free non-asymptotic risk bound for the expected error of the
proposed estimator. It is the first result of this kind in the literature and
involves only the effective rank of the covariance matrix. Finally, we show
that the obtained results can be extended to sub-Gaussian distributions, as
well as to the cases of unknown rate of contamination or unknown covariance
matrix.Comment: 41 pages, 5 figures; added sub-Gaussian case with unknown Sigma or
ep
Mixture extreme learning machine algorithm for robust regression
The extreme learning machine (ELM) is a well-known approach for training single hidden layer feedforward neural networks (SLFNs) in machine learning. However, ELM is most effective when used for regression on datasets with simple Gaussian distributed error because it often employs a squared loss in its objective function. In contrast, real-world data is often collected from unpredictable and diverse contexts, which may contain complex noise that cannot be characterized by a single distribution. To address this challenge, we propose a robust mixture ELM algorithm, called Mixture-ELM, that enhances modeling capability and resilience to both Gaussian and non-Gaussian noise. The Mixture-ELM algorithm uses an adjusted objective function that blends Gaussian and Laplacian distributions to approximate any continuous distribution and match the noise. The Gaussian mixture accurately models the residual distribution, while the inclusion of the Laplacian distribution addresses the limitations of the Gaussian distribution in identifying outliers. We derive a solution to the novel objective function using the expectation maximization (EM) and iteratively reweighted least squares (IRLS) algorithms. We evaluate the effectiveness of the algorithm through numerical simulation and experiments on benchmark datasets, thereby demonstrating its superiority over other state-of-the-art machine learning methods in terms of robustness and generalization
Learning how to be robust: Deep polynomial regression
Polynomial regression is a recurrent problem with a large number of
applications. In computer vision it often appears in motion analysis. Whatever
the application, standard methods for regression of polynomial models tend to
deliver biased results when the input data is heavily contaminated by outliers.
Moreover, the problem is even harder when outliers have strong structure.
Departing from problem-tailored heuristics for robust estimation of parametric
models, we explore deep convolutional neural networks. Our work aims to find a
generic approach for training deep regression models without the explicit need
of supervised annotation. We bypass the need for a tailored loss function on
the regression parameters by attaching to our model a differentiable hard-wired
decoder corresponding to the polynomial operation at hand. We demonstrate the
value of our findings by comparing with standard robust regression methods.
Furthermore, we demonstrate how to use such models for a real computer vision
problem, i.e., video stabilization. The qualitative and quantitative
experiments show that neural networks are able to learn robustness for general
polynomial regression, with results that well overpass scores of traditional
robust estimation methods.Comment: 18 pages, conferenc
Robust Sparse Canonical Correlation Analysis
Canonical correlation analysis (CCA) is a multivariate statistical method
which describes the associations between two sets of variables. The objective
is to find linear combinations of the variables in each data set having maximal
correlation. This paper discusses a method for Robust Sparse CCA. Sparse
estimation produces canonical vectors with some of their elements estimated as
exactly zero. As such, their interpretability is improved. We also robustify
the method such that it can cope with outliers in the data. To estimate the
canonical vectors, we convert the CCA problem into an alternating regression
framework, and use the sparse Least Trimmed Squares estimator. We illustrate
the good performance of the Robust Sparse CCA method in several simulation
studies and two real data examples
Robust computation of linear models by convex relaxation
Consider a dataset of vector-valued observations that consists of noisy
inliers, which are explained well by a low-dimensional subspace, along with
some number of outliers. This work describes a convex optimization problem,
called REAPER, that can reliably fit a low-dimensional model to this type of
data. This approach parameterizes linear subspaces using orthogonal projectors,
and it uses a relaxation of the set of orthogonal projectors to reach the
convex formulation. The paper provides an efficient algorithm for solving the
REAPER problem, and it documents numerical experiments which confirm that
REAPER can dependably find linear structure in synthetic and natural data. In
addition, when the inliers lie near a low-dimensional subspace, there is a
rigorous theory that describes when REAPER can approximate this subspace.Comment: Formerly titled "Robust computation of linear models, or How to find
a needle in a haystack
Robust variable screening for regression using factor profiling
Sure Independence Screening is a fast procedure for variable selection in
ultra-high dimensional regression analysis. Unfortunately, its performance
greatly deteriorates with increasing dependence among the predictors. To solve
this issue, Factor Profiled Sure Independence Screening (FPSIS) models the
correlation structure of the predictor variables, assuming that it can be
represented by a few latent factors. The correlations can then be profiled out
by projecting the data onto the orthogonal complement of the subspace spanned
by these factors. However, neither of these methods can handle the presence of
outliers in the data. Therefore, we propose a robust screening method which
uses a least trimmed squares method to estimate the latent factors and the
factor profiled variables. Variable screening is then performed on factor
profiled variables by using regression MM-estimators. Different types of
outliers in this model and their roles in variable screening are studied. Both
simulation studies and a real data analysis show that the proposed robust
procedure has good performance on clean data and outperforms the two nonrobust
methods on contaminated data
- …