1,402 research outputs found
A survey of sparse representation: algorithms and applications
Sparse representation has attracted much attention from researchers in fields
of signal processing, image processing, computer vision and pattern
recognition. Sparse representation also has a good reputation in both
theoretical research and practical applications. Many different algorithms have
been proposed for sparse representation. The main purpose of this article is to
provide a comprehensive study and an updated review on sparse representation
and to supply a guidance for researchers. The taxonomy of sparse representation
methods can be studied from various viewpoints. For example, in terms of
different norm minimizations used in sparsity constraints, the methods can be
roughly categorized into five groups: sparse representation with -norm
minimization, sparse representation with -norm (0p1) minimization,
sparse representation with -norm minimization and sparse representation
with -norm minimization. In this paper, a comprehensive overview of
sparse representation is provided. The available sparse representation
algorithms can also be empirically categorized into four groups: greedy
strategy approximation, constrained optimization, proximity algorithm-based
optimization, and homotopy algorithm-based sparse representation. The
rationales of different algorithms in each category are analyzed and a wide
range of sparse representation applications are summarized, which could
sufficiently reveal the potential nature of the sparse representation theory.
Specifically, an experimentally comparative study of these sparse
representation algorithms was presented. The Matlab code used in this paper can
be available at: http://www.yongxu.org/lunwen.html.Comment: Published on IEEE Access, Vol. 3, pp. 490-530, 201
Differentiable Linearized ADMM
Recently, a number of learning-based optimization methods that combine
data-driven architectures with the classical optimization algorithms have been
proposed and explored, showing superior empirical performance in solving
various ill-posed inverse problems, but there is still a scarcity of rigorous
analysis about the convergence behaviors of learning-based optimization. In
particular, most existing analyses are specific to unconstrained problems but
cannot apply to the more general cases where some variables of interest are
subject to certain constraints. In this paper, we propose Differentiable
Linearized ADMM (D-LADMM) for solving the problems with linear constraints.
Specifically, D-LADMM is a K-layer LADMM inspired deep neural network, which is
obtained by firstly introducing some learnable weights in the classical
Linearized ADMM algorithm and then generalizing the proximal operator to some
learnable activation function. Notably, we rigorously prove that there exist a
set of learnable parameters for D-LADMM to generate globally converged
solutions, and we show that those desired parameters can be attained by
training D-LADMM in a proper way. To the best of our knowledge, we are the
first to provide the convergence analysis for the learning-based optimization
method on constrained problems.Comment: Accepted by ICML201
Optimized Structured Sparse Sensing Matrices for Compressive Sensing
We consider designing a robust structured sparse sensing matrix consisting of
a sparse matrix with a few non-zero entries per row and a dense base matrix for
capturing signals efficiently We design the robust structured sparse sensing
matrix through minimizing the distance between the Gram matrix of the
equivalent dictionary and the target Gram of matrix holding small mutual
coherence. Moreover, a regularization is added to enforce the robustness of the
optimized structured sparse sensing matrix to the sparse representation error
(SRE) of signals of interests. An alternating minimization algorithm with
global sequence convergence is proposed for solving the corresponding
optimization problem. Numerical experiments on synthetic data and natural
images show that the obtained structured sensing matrix results in a higher
signal reconstruction than a random dense sensing matrix.Comment: 2 tables, 10 figure
A Unified Algorithmic Framework for Block-Structured Optimization Involving Big Data
This article presents a powerful algorithmic framework for big data
optimization, called the Block Successive Upper bound Minimization (BSUM). The
BSUM includes as special cases many well-known methods for analyzing massive
data sets, such as the Block Coordinate Descent (BCD), the Convex-Concave
Procedure (CCCP), the Block Coordinate Proximal Gradient (BCPG) method, the
Nonnegative Matrix Factorization (NMF), the Expectation Maximization (EM)
method and so on. In this article, various features and properties of the BSUM
are discussed from the viewpoint of design flexibility, computational
efficiency, parallel/distributed implementation and the required communication
overhead. Illustrative examples from networking, signal processing and machine
learning are presented to demonstrate the practical performance of the BSUM
framewor
Variations on the CSC model
Over the past decade, the celebrated sparse representation model has achieved
impressive results in various signal and image processing tasks. A
convolutional version of this model, termed convolutional sparse coding (CSC),
has been recently reintroduced and extensively studied. CSC brings a natural
remedy to the limitation of typical sparse enforcing approaches of handling
global and high-dimensional signals by local, patch-based, processing. While
the classic field of sparse representations has been able to cater for the
diverse challenges of different signal processing tasks by considering a wide
range of problem formulations, almost all available algorithms that deploy the
CSC model consider the same problem form. As we argue in this
paper, this CSC pursuit formulation is also too restrictive as it fails to
explicitly exploit some local characteristics of the signal. This work expands
the range of formulations for the CSC model by proposing two convex
alternatives that merge global norms with local penalties and constraints. The
main contribution of this work is the derivation of efficient and provably
converging algorithms to solve these new sparse coding formulations
Efficient Sum of Outer Products Dictionary Learning (SOUP-DIL) and Its Application to Inverse Problems
The sparsity of signals in a transform domain or dictionary has been
exploited in applications such as compression, denoising and inverse problems.
More recently, data-driven adaptation of synthesis dictionaries has shown
promise compared to analytical dictionary models. However, dictionary learning
problems are typically non-convex and NP-hard, and the usual alternating
minimization approaches for these problems are often computationally expensive,
with the computations dominated by the NP-hard synthesis sparse coding step.
This paper exploits the ideas that drive algorithms such as K-SVD, and
investigates in detail efficient methods for aggregate sparsity penalized
dictionary learning by first approximating the data with a sum of sparse
rank-one matrices (outer products) and then using a block coordinate descent
approach to estimate the unknowns. The resulting block coordinate descent
algorithms involve efficient closed-form solutions. Furthermore, we consider
the problem of dictionary-blind image reconstruction, and propose novel and
efficient algorithms for adaptive image reconstruction using block coordinate
descent and sum of outer products methodologies. We provide a convergence study
of the algorithms for dictionary learning and dictionary-blind image
reconstruction. Our numerical experiments show the promising performance and
speed-ups provided by the proposed methods over previous schemes in sparse data
representation and compressed sensing-based image reconstruction.Comment: Accepted to IEEE Transactions on Computational Imaging. This paper
also cites experimental results reported in arXiv:1511.0884
Learning a collaborative multiscale dictionary based on robust empirical mode decomposition
Dictionary learning is a challenge topic in many image processing areas. The
basic goal is to learn a sparse representation from an overcomplete basis set.
Due to combining the advantages of generic multiscale representations with
learning based adaptivity, multiscale dictionary representation approaches have
the power in capturing structural characteristics of natural images. However,
existing multiscale learning approaches still suffer from three main
weaknesses: inadaptability to diverse scales of image data, sensitivity to
noise and outliers, difficulty to determine optimal dictionary structure. In
this paper, we present a novel multiscale dictionary learning paradigm for
sparse image representations based on an improved empirical mode decomposition.
This powerful data-driven analysis tool for multi-dimensional signal can fully
adaptively decompose the image into multiscale oscillating components according
to intrinsic modes of data self. This treatment can obtain a robust and
effective sparse representation, and meanwhile generates a raw base dictionary
at multiple geometric scales and spatial frequency bands. This dictionary is
refined by selecting optimal oscillating atoms based on frequency clustering.
In order to further enhance sparsity and generalization, a tolerance dictionary
is learned using a coherence regularized model. A fast proximal scheme is
developed to optimize this model. The multiscale dictionary is considered as
the product of oscillating dictionary and tolerance dictionary. Experimental
results demonstrate that the proposed learning approach has the superior
performance in sparse image representations as compared with several competing
methods. We also show the promising results in image denoising application.Comment: to be published in Neurocomputin
ADMM for Multiaffine Constrained Optimization
We expand the scope of the alternating direction method of multipliers
(ADMM). Specifically, we show that ADMM, when employed to solve problems with
multiaffine constraints that satisfy certain verifiable assumptions, converges
to the set of constrained stationary points if the penalty parameter in the
augmented Lagrangian is sufficiently large. When the Kurdyka-\L{}ojasiewicz
(K-\L{}) property holds, this is strengthened to convergence to a single
constrained stationary point. Our analysis applies under assumptions that we
have endeavored to make as weak as possible. It applies to problems that
involve nonconvex and/or nonsmooth objective terms, in addition to the
multiaffine constraints that can involve multiple (three or more) blocks of
variables. To illustrate the applicability of our results, we describe examples
including nonnegative matrix factorization, sparse learning, risk parity
portfolio selection, nonconvex formulations of convex problems, and neural
network training. In each case, our ADMM approach encounters only subproblems
that have closed-form solutions.Comment: v3: 37 pages, 7 figures v2: 32 pages, 0 figures. v1: 26 pages, 0
figure
Proximal methods for the latent group lasso penalty
We consider a regularized least squares problem, with regularization by
structured sparsity-inducing norms, which extend the usual and the
group lasso penalty, by allowing the subsets to overlap. Such regularizations
lead to nonsmooth problems that are difficult to optimize, and we propose in
this paper a suitable version of an accelerated proximal method to solve them.
We prove convergence of a nested procedure, obtained composing an accelerated
proximal method with an inner algorithm for computing the proximity operator.
By exploiting the geometrical properties of the penalty, we devise a new active
set strategy, thanks to which the inner iteration is relatively fast, thus
guaranteeing good computational performances of the overall algorithm. Our
approach allows to deal with high dimensional problems without pre-processing
for dimensionality reduction, leading to better computational and prediction
performances with respect to the state-of-the art methods, as shown empirically
both on toy and real data.Comment: 4 figure
Decomposition into Low-rank plus Additive Matrices for Background/Foreground Separation: A Review for a Comparative Evaluation with a Large-Scale Dataset
Recent research on problem formulations based on decomposition into low-rank
plus sparse matrices shows a suitable framework to separate moving objects from
the background. The most representative problem formulation is the Robust
Principal Component Analysis (RPCA) solved via Principal Component Pursuit
(PCP) which decomposes a data matrix in a low-rank matrix and a sparse matrix.
However, similar robust implicit or explicit decompositions can be made in the
following problem formulations: Robust Non-negative Matrix Factorization
(RNMF), Robust Matrix Completion (RMC), Robust Subspace Recovery (RSR), Robust
Subspace Tracking (RST) and Robust Low-Rank Minimization (RLRM). The main goal
of these similar problem formulations is to obtain explicitly or implicitly a
decomposition into low-rank matrix plus additive matrices. In this context,
this work aims to initiate a rigorous and comprehensive review of the similar
problem formulations in robust subspace learning and tracking based on
decomposition into low-rank plus additive matrices for testing and ranking
existing algorithms for background/foreground separation. For this, we first
provide a preliminary review of the recent developments in the different
problem formulations which allows us to define a unified view that we called
Decomposition into Low-rank plus Additive Matrices (DLAM). Then, we examine
carefully each method in each robust subspace learning/tracking frameworks with
their decomposition, their loss functions, their optimization problem and their
solvers. Furthermore, we investigate if incremental algorithms and real-time
implementations can be achieved for background/foreground separation. Finally,
experimental results on a large-scale dataset called Background Models
Challenge (BMC 2012) show the comparative performance of 32 different robust
subspace learning/tracking methods.Comment: 121 pages, 5 figures, submitted to Computer Science Review. arXiv
admin note: text overlap with arXiv:1312.7167, arXiv:1109.6297,
arXiv:1207.3438, arXiv:1105.2126, arXiv:1404.7592, arXiv:1210.0805,
arXiv:1403.8067 by other authors, Computer Science Review, November 201
- …