11 research outputs found
Smoothing algorithms for nonsmooth and nonconvex minimization over the stiefel manifold
We consider a class of nonsmooth and nonconvex optimization problems over the
Stiefel manifold where the objective function is the summation of a nonconvex
smooth function and a nonsmooth Lipschitz continuous convex function composed
with an linear mapping. We propose three numerical algorithms for solving this
problem, by combining smoothing methods and some existing algorithms for smooth
optimization over the Stiefel manifold. In particular, we approximate the
aforementioned nonsmooth convex function by its Moreau envelope in our
smoothing methods, and prove that the Moreau envelope has many favorable
properties. Thanks to this and the scheme for updating the smoothing parameter,
we show that any accumulation point of the solution sequence generated by the
proposed algorithms is a stationary point of the original optimization problem.
Numerical experiments on building graph Fourier basis are conducted to
demonstrate the efficiency of the proposed algorithms.Comment: 22 page
Sparse PCA: Optimal rates and adaptive estimation
Principal component analysis (PCA) is one of the most commonly used
statistical procedures with a wide range of applications. This paper considers
both minimax and adaptive estimation of the principal subspace in the high
dimensional setting. Under mild technical conditions, we first establish the
optimal rates of convergence for estimating the principal subspace which are
sharp with respect to all the parameters, thus providing a complete
characterization of the difficulty of the estimation problem in term of the
convergence rate. The lower bound is obtained by calculating the local metric
entropy and an application of Fano's lemma. The rate optimal estimator is
constructed using aggregation, which, however, might not be computationally
feasible. We then introduce an adaptive procedure for estimating the principal
subspace which is fully data driven and can be computed efficiently. It is
shown that the estimator attains the optimal rates of convergence
simultaneously over a large collection of the parameter spaces. A key idea in
our construction is a reduction scheme which reduces the sparse PCA problem to
a high-dimensional multivariate regression problem. This method is potentially
also useful for other related problems.Comment: Published in at http://dx.doi.org/10.1214/13-AOS1178 the Annals of
Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical
Statistics (http://www.imstat.org
Covariance Eigenvector Sparsity for Compression and Denoising
Sparsity in the eigenvectors of signal covariance matrices is exploited in
this paper for compression and denoising. Dimensionality reduction (DR) and
quantization modules present in many practical compression schemes such as
transform codecs, are designed to capitalize on this form of sparsity and
achieve improved reconstruction performance compared to existing
sparsity-agnostic codecs. Using training data that may be noisy a novel
sparsity-aware linear DR scheme is developed to fully exploit sparsity in the
covariance eigenvectors and form noise-resilient estimates of the principal
covariance eigenbasis. Sparsity is effected via norm-one regularization, and
the associated minimization problems are solved using computationally efficient
coordinate descent iterations. The resulting eigenspace estimator is shown
capable of identifying a subset of the unknown support of the eigenspace basis
vectors even when the observation noise covariance matrix is unknown, as long
as the noise power is sufficiently low. It is proved that the sparsity-aware
estimator is asymptotically normal, and the probability to correctly identify
the signal subspace basis support approaches one, as the number of training
data grows large. Simulations using synthetic data and images, corroborate that
the proposed algorithms achieve improved reconstruction quality relative to
alternatives.Comment: IEEE Transcations on Signal Processing, 2012 (to appear
State–of–the–art report on nonlinear representation of sources and channels
This report consists of two complementary parts, related to the modeling of two important sources of nonlinearities in a communications system. In the first part, an overview of important past work related to the estimation, compression and processing of sparse data through the use of nonlinear models is provided. In the second part, the current state of the art on the representation of wireless channels in the presence of nonlinearities is summarized. In addition to the characteristics of the nonlinear wireless fading channel, some information is also provided on recent approaches to the sparse representation of such channels
Sparse principal component analysis and iterative thresholding
Principal component analysis (PCA) is a classical dimension reduction method
which projects data onto the principal subspace spanned by the leading
eigenvectors of the covariance matrix. However, it behaves poorly when the
number of features p is comparable to, or even much larger than, the sample
size n. In this paper, we propose a new iterative thresholding approach for
estimating principal subspaces in the setting where the leading eigenvectors
are sparse. Under a spiked covariance model, we find that the new approach
recovers the principal subspace and leading eigenvectors consistently, and even
optimally, in a range of high-dimensional sparse settings. Simulated examples
also demonstrate its competitive performance.Comment: Published in at http://dx.doi.org/10.1214/13-AOS1097 the Annals of
Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical
Statistics (http://www.imstat.org
Linear dimensionality reduction: Survey, insights, and generalizations
Linear dimensionality reduction methods are a cornerstone of analyzing high
dimensional data, due to their simple geometric interpretations and typically
attractive computational properties. These methods capture many data features
of interest, such as covariance, dynamical structure, correlation between data
sets, input-output relationships, and margin between data classes. Methods have
been developed with a variety of names and motivations in many fields, and
perhaps as a result the connections between all these methods have not been
highlighted. Here we survey methods from this disparate literature as
optimization programs over matrix manifolds. We discuss principal component
analysis, factor analysis, linear multidimensional scaling, Fisher's linear
discriminant analysis, canonical correlations analysis, maximum autocorrelation
factors, slow feature analysis, sufficient dimensionality reduction,
undercomplete independent component analysis, linear regression, distance
metric learning, and more. This optimization framework gives insight to some
rarely discussed shortcomings of well-known methods, such as the suboptimality
of certain eigenvector solutions. Modern techniques for optimization over
matrix manifolds enable a generic linear dimensionality reduction solver, which
accepts as input data and an objective to be optimized, and returns, as output,
an optimal low-dimensional projection of the data. This simple optimization
framework further allows straightforward generalizations and novel variants of
classical methods, which we demonstrate here by creating an
orthogonal-projection canonical correlations analysis. More broadly, this
survey and generic solver suggest that linear dimensionality reduction can move
toward becoming a blackbox, objective-agnostic numerical technology.JPC and ZG received funding from the UK Engineering and Physical Sciences Research Council (EPSRC EP/H019472/1). JPC received funding from a Sloan Research Fellowship, the Simons Foundation (SCGB#325171 and SCGB#325233), the Grossman Center at Columbia University, and the Gatsby Charitable Trust.This is the author accepted manuscript. The final version is available from MIT Press via http://jmlr.org/papers/v16/cunningham15a.htm
Advanced sequential Monte Carlo methods and their applications to sparse sensor network for detection and estimation
The general state space models present a flexible framework for modeling dynamic systems and therefore have vast applications in many disciplines such as engineering, economics, biology, etc. However, optimal estimation problems of non-linear non-Gaussian state space models are analytically intractable in general. Sequential Monte Carlo (SMC) methods become a very popular class of simulation-based methods for the solution of optimal estimation problems. The advantages of SMC methods in comparison with classical filtering methods such as Kalman Filter and Extended Kalman Filter are that they are able to handle non-linear non-Gaussian scenarios without relying on any local linearization techniques. In this thesis, we present an advanced SMC method and the study of its asymptotic behavior. We apply the proposed SMC method in a target tracking problem using different observation models. Specifically, a distributed SMC algorithm is developed for a wireless sensor network (WSN) that incorporates with an informative-sensor detection technique. The novel SMC algorithm is designed to surmount the degeneracy problem by employing a multilevel Markov chain Monte Carlo (MCMC) procedure constructed by engaging drift homotopy and likelihood bridging techniques. The observations are gathered only from the informative sensors, which are sensing useful observations of the nearby moving targets. The detection of those informative sensors, which are typically a small portion of the WSN, is taking place by using a sparsity-aware matrix decomposition technique. Simulation results showcase that our algorithm outperforms current popular tracking algorithms such as bootstrap filter and auxiliary particle filter in many scenarios