1,609 research outputs found
A Flexible and Efficient Algorithmic Framework for Constrained Matrix and Tensor Factorization
We propose a general algorithmic framework for constrained matrix and tensor
factorization, which is widely used in signal processing and machine learning.
The new framework is a hybrid between alternating optimization (AO) and the
alternating direction method of multipliers (ADMM): each matrix factor is
updated in turn, using ADMM, hence the name AO-ADMM. This combination can
naturally accommodate a great variety of constraints on the factor matrices,
and almost all possible loss measures for the fitting. Computation caching and
warm start strategies are used to ensure that each update is evaluated
efficiently, while the outer AO framework exploits recent developments in block
coordinate descent (BCD)-type methods which help ensure that every limit point
is a stationary point, as well as faster and more robust convergence in
practice. Three special cases are studied in detail: non-negative matrix/tensor
factorization, constrained matrix/tensor completion, and dictionary learning.
Extensive simulations and experiments with real data are used to showcase the
effectiveness and broad applicability of the proposed framework
ADMM for Multiaffine Constrained Optimization
We expand the scope of the alternating direction method of multipliers
(ADMM). Specifically, we show that ADMM, when employed to solve problems with
multiaffine constraints that satisfy certain verifiable assumptions, converges
to the set of constrained stationary points if the penalty parameter in the
augmented Lagrangian is sufficiently large. When the Kurdyka-\L{}ojasiewicz
(K-\L{}) property holds, this is strengthened to convergence to a single
constrained stationary point. Our analysis applies under assumptions that we
have endeavored to make as weak as possible. It applies to problems that
involve nonconvex and/or nonsmooth objective terms, in addition to the
multiaffine constraints that can involve multiple (three or more) blocks of
variables. To illustrate the applicability of our results, we describe examples
including nonnegative matrix factorization, sparse learning, risk parity
portfolio selection, nonconvex formulations of convex problems, and neural
network training. In each case, our ADMM approach encounters only subproblems
that have closed-form solutions.Comment: v3: 37 pages, 7 figures v2: 32 pages, 0 figures. v1: 26 pages, 0
figure
Global hard thresholding algorithms for joint sparse image representation and denoising
Sparse coding of images is traditionally done by cutting them into small
patches and representing each patch individually over some dictionary given a
pre-determined number of nonzero coefficients to use for each patch. In lack of
a way to effectively distribute a total number (or global budget) of nonzero
coefficients across all patches, current sparse recovery algorithms distribute
the global budget equally across all patches despite the wide range of
differences in structural complexity among them. In this work we propose a new
framework for joint sparse representation and recovery of all image patches
simultaneously. We also present two novel global hard thresholding algorithms,
based on the notion of variable splitting, for solving the joint sparse model.
Experimentation using both synthetic and real data shows effectiveness of the
proposed framework for sparse image representation and denoising tasks.
Additionally, time complexity analysis of the proposed algorithms indicate high
scalability of both algorithms, making them favorable to use on large megapixel
images
Linearized ADMM for Non-convex Non-smooth Optimization with Convergence Analysis
Linearized alternating direction method of multipliers (ADMM) as an extension
of ADMM has been widely used to solve linearly constrained problems in signal
processing, machine leaning, communications, and many other fields. Despite its
broad applications in nonconvex optimization, for a great number of nonconvex
and nonsmooth objective functions, its theoretical convergence guarantee is
still an open problem. In this paper, we propose a two-block linearized ADMM
and a multi-block parallel linearized ADMM for problems with nonconvex and
nonsmooth objectives. Mathematically, we present that the algorithms can
converge for a broader class of objective functions under less strict
assumptions compared with previous works. Furthermore, our proposed algorithm
can update coupled variables in parallel and work for less restrictive
nonconvex problems, where the traditional ADMM may have difficulties in solving
subproblems.Comment: 29 pages, 2 tables, 2 figure
Fast Convolutional Sparse Coding in the Dual Domain
Convolutional sparse coding (CSC) is an important building block of many
computer vision applications ranging from image and video compression to deep
learning. We present two contributions to the state of the art in CSC. First,
we significantly speed up the computation by proposing a new optimization
framework that tackles the problem in the dual domain. Second, we extend the
original formulation to higher dimensions in order to process a wider range of
inputs, such as RGB images and videos. Our results show up to 20 times speedup
compared to current state-of-the-art CSC solvers
A Fast and Efficient Algorithm for Reconstructing MR images From Partial Fourier Samples
In this paper, the problem of Magnetic Resonance (MR) image reconstruction
from partial Fourier samples has been considered. To this aim, we leverage the
evidence that MR images are sparser than their zero-filled reconstructed ones
from incomplete Fourier samples. This information can be used to define an
optimization problem which searches for the sparsest possible image conforming
with the available Fourier samples. We solve the resulting problem using the
well-known Alternating Direction Method of Multipliers (ADMM). Unlike most
existing methods that work with small over-lapping image patches, the proposed
algorithm considers the whole image without dividing it into small blocks.
Experimental results prominently confirm its promising performance and
advantages over the existing methods
LSALSA: Accelerated Source Separation via Learned Sparse Coding
We propose an efficient algorithm for the generalized sparse coding (SC)
inference problem. The proposed framework applies to both the single dictionary
setting, where each data point is represented as a sparse combination of the
columns of one dictionary matrix, as well as the multiple dictionary setting as
given in morphological component analysis (MCA), where the goal is to separate
a signal into additive parts such that each part has distinct sparse
representation within a corresponding dictionary. Both the SC task and its
generalization via MCA have been cast as -regularized least-squares
optimization problems. To accelerate traditional acquisition of sparse codes,
we propose a deep learning architecture that constitutes a trainable
time-unfolded version of the Split Augmented Lagrangian Shrinkage Algorithm
(SALSA), a special case of the Alternating Direction Method of Multipliers
(ADMM). We empirically validate both variants of the algorithm, that we refer
to as LSALSA (learned-SALSA), on image vision tasks and demonstrate that at
inference our networks achieve vast improvements in terms of the running time,
the quality of estimated sparse codes, and visual clarity on both classic SC
and MCA problems. Finally, we present a theoretical framework for analyzing
LSALSA network: we show that the proposed approach exactly implements a
truncated ADMM applied to a new, learned cost function with curvature modified
by one of the learned parameterized matrices. We extend a very recent
Stochastic Alternating Optimization analysis framework to show that a gradient
descent step along this learned loss landscape is equivalent to a modified
gradient descent step along the original loss landscape. In this framework, the
acceleration achieved by LSALSA could potentially be explained by the network's
ability to learn a correction to the gradient direction of steeper descent.Comment: ECML-PKDD 2019 via journal track; Special Issue Mach Learn (2019
Tomographic Image Reconstruction using Training images
We describe and examine an algorithm for tomographic image reconstruction
where prior knowledge about the solution is available in the form of training
images. We first construct a nonnegative dictionary based on prototype elements
from the training images; this problem is formulated as a regularized
non-negative matrix factorization. Incorporating the dictionary as a prior in a
convex reconstruction problem, we then find an approximate solution with a
sparse representation in the dictionary. The dictionary is applied to
non-overlapping patches of the image, which reduces the computational
complexity compared to other algorithms. Computational experiments clarify the
choice and interplay of the model parameters and the regularization parameters,
and we show that in few-projection low-dose settings our algorithm is
competitive with total variation regularization and tends to include more
texture and more correct edges.Comment: 25 pages, 12 figure
Penalty Dual Decomposition Method For Nonsmooth Nonconvex Optimization
Many contemporary signal processing, machine learning and wireless
communication applications can be formulated as nonconvex nonsmooth
optimization problems. Often there is a lack of efficient algorithms for these
problems, especially when the optimization variables are nonlinearly coupled in
some nonconvex constraints. In this work, we propose an algorithm named penalty
dual decomposition (PDD) for these difficult problems and discuss its various
applications. The PDD is a double-loop iterative algorithm. Its inner
iterations is used to inexactly solve a nonconvex nonsmooth augmented
Lagrangian problem via block-coordinate-descenttype methods, while its outer
iteration updates the dual variables and/or a penalty parameter. In Part I of
this work, we describe the PDD algorithm and rigorously establish its
convergence to KKT solutions. In Part II we evaluate the performance of PDD by
customizing it to three applications arising from signal processing and
wireless communications.Comment: Two part paper, 27 figure
Noisy Matrix Completion under Sparse Factor Models
This paper examines a general class of noisy matrix completion tasks where
the goal is to estimate a matrix from observations obtained at a subset of its
entries, each of which is subject to random noise or corruption. Our specific
focus is on settings where the matrix to be estimated is well-approximated by a
product of two (a priori unknown) matrices, one of which is sparse. Such
structural models - referred to here as "sparse factor models" - have been
widely used, for example, in subspace clustering applications, as well as in
contemporary sparse modeling and dictionary learning tasks. Our main
theoretical contributions are estimation error bounds for sparsity-regularized
maximum likelihood estimators for problems of this form, which are applicable
to a number of different observation noise or corruption models. Several
specific implications are examined, including scenarios where observations are
corrupted by additive Gaussian noise or additive heavier-tailed (Laplace)
noise, Poisson-distributed observations, and highly-quantized (e.g., one-bit)
observations. We also propose a simple algorithmic approach based on the
alternating direction method of multipliers for these tasks, and provide
experimental evidence to support our error analyses.Comment: 42 Pages, 7 Figures, Submitted to IEEE Transactions on Information
Theor
- β¦