18,830 research outputs found
Sparse Approximation via Penalty Decomposition Methods
In this paper we consider sparse approximation problems, that is, general
minimization problems with the -"norm" of a vector being a part of
constraints or objective function. In particular, we first study the
first-order optimality conditions for these problems. We then propose penalty
decomposition (PD) methods for solving them in which a sequence of penalty
subproblems are solved by a block coordinate descent (BCD) method. Under some
suitable assumptions, we establish that any accumulation point of the sequence
generated by the PD methods satisfies the first-order optimality conditions of
the problems. Furthermore, for the problems in which the part is the only
nonconvex part, we show that such an accumulation point is a local minimizer of
the problems. In addition, we show that any accumulation point of the sequence
generated by the BCD method is a saddle point of the penalty subproblem.
Moreover, for the problems in which the part is the only nonconvex part,
we establish that such an accumulation point is a local minimizer of the
penalty subproblem. Finally, we test the performance of our PD methods by
applying them to sparse logistic regression, sparse inverse covariance
selection, and compressed sensing problems. The computational results
demonstrate that our methods generally outperform the existing methods in terms
of solution quality and/or speed.Comment: 31 pages, 3 figures and 9 tables. arXiv admin note: substantial text
overlap with arXiv:1008.537
Sparse permutation invariant covariance estimation
The paper proposes a method for constructing a sparse estimator for the
inverse covariance (concentration) matrix in high-dimensional settings. The
estimator uses a penalized normal likelihood approach and forces sparsity by
using a lasso-type penalty. We establish a rate of convergence in the Frobenius
norm as both data dimension and sample size are allowed to grow, and
show that the rate depends explicitly on how sparse the true concentration
matrix is. We also show that a correlation-based version of the method exhibits
better rates in the operator norm. We also derive a fast iterative algorithm
for computing the estimator, which relies on the popular Cholesky decomposition
of the inverse but produces a permutation-invariant estimator. The method is
compared to other estimators on simulated data and on a real data example of
tumor tissue classification using gene expression data.Comment: Published in at http://dx.doi.org/10.1214/08-EJS176 the Electronic
Journal of Statistics (http://www.i-journals.org/ejs/) by the Institute of
Mathematical Statistics (http://www.imstat.org
A successive difference-of-convex approximation method for a class of nonconvex nonsmooth optimization problems
We consider a class of nonconvex nonsmooth optimization problems whose
objective is the sum of a smooth function and a finite number of nonnegative
proper closed possibly nonsmooth functions (whose proximal mappings are easy to
compute), some of which are further composed with linear maps. This kind of
problems arises naturally in various applications when different regularizers
are introduced for inducing simultaneous structures in the solutions. Solving
these problems, however, can be challenging because of the coupled nonsmooth
functions: the corresponding proximal mapping can be hard to compute so that
standard first-order methods such as the proximal gradient algorithm cannot be
applied efficiently. In this paper, we propose a successive
difference-of-convex approximation method for solving this kind of problems. In
this algorithm, we approximate the nonsmooth functions by their Moreau
envelopes in each iteration. Making use of the simple observation that Moreau
envelopes of nonnegative proper closed functions are continuous {\em
difference-of-convex} functions, we can then approximately minimize the
approximation function by first-order methods with suitable majorization
techniques. These first-order methods can be implemented efficiently thanks to
the fact that the proximal mapping of {\em each} nonsmooth function is easy to
compute. Under suitable assumptions, we prove that the sequence generated by
our method is bounded and any accumulation point is a stationary point of the
objective. We also discuss how our method can be applied to concrete
applications such as nonconvex fused regularized optimization problems and
simultaneously structured matrix optimization problems, and illustrate the
performance numerically for these two specific applications
Learning incoherent dictionaries for sparse approximation using iterative projections and rotations
This work was supported by the Queen Mary University of London School Studentship, the EU FET-Open project FP7-
ICT-225913-SMALL. Sparse Models, Algorithms and Learning for Large-scale data and a Leadership Fellowship from the UK
Engineering and Physical Sciences Research Council (EPSRC)
Using Underapproximations for Sparse Nonnegative Matrix Factorization
Nonnegative Matrix Factorization consists in (approximately) factorizing a
nonnegative data matrix by the product of two low-rank nonnegative matrices. It
has been successfully applied as a data analysis technique in numerous domains,
e.g., text mining, image processing, microarray data analysis, collaborative
filtering, etc.
We introduce a novel approach to solve NMF problems, based on the use of an
underapproximation technique, and show its effectiveness to obtain sparse
solutions. This approach, based on Lagrangian relaxation, allows the resolution
of NMF problems in a recursive fashion. We also prove that the
underapproximation problem is NP-hard for any fixed factorization rank, using a
reduction of the maximum edge biclique problem in bipartite graphs.
We test two variants of our underapproximation approach on several standard
image datasets and show that they provide sparse part-based representations
with low reconstruction error. Our results are comparable and sometimes
superior to those obtained by two standard Sparse Nonnegative Matrix
Factorization techniques.Comment: Version 2 removed the section about convex reformulations, which was
not central to the development of our main results; added material to the
introduction; added a review of previous related work (section 2.3);
completely rewritten the last part (section 4) to provide extensive numerical
results supporting our claims. Accepted in J. of Pattern Recognitio
Transposable regularized covariance models with an application to missing data imputation
Missing data estimation is an important challenge with high-dimensional data
arranged in the form of a matrix. Typically this data matrix is transposable,
meaning that either the rows, columns or both can be treated as features. To
model transposable data, we present a modification of the matrix-variate
normal, the mean-restricted matrix-variate normal, in which the rows and
columns each have a separate mean vector and covariance matrix. By placing
additive penalties on the inverse covariance matrices of the rows and columns,
these so-called transposable regularized covariance models allow for maximum
likelihood estimation of the mean and nonsingular covariance matrices. Using
these models, we formulate EM-type algorithms for missing data imputation in
both the multivariate and transposable frameworks. We present theoretical
results exploiting the structure of our transposable models that allow these
models and imputation methods to be applied to high-dimensional data.
Simulations and results on microarray data and the Netflix data show that these
imputation techniques often outperform existing methods and offer a greater
degree of flexibility.Comment: Published in at http://dx.doi.org/10.1214/09-AOAS314 the Annals of
Applied Statistics (http://www.imstat.org/aoas/) by the Institute of
Mathematical Statistics (http://www.imstat.org
- …