94,750 research outputs found
Dual-Space Analysis of the Sparse Linear Model
Sparse linear (or generalized linear) models combine a standard likelihood
function with a sparse prior on the unknown coefficients. These priors can
conveniently be expressed as a maximization over zero-mean Gaussians with
different variance hyperparameters. Standard MAP estimation (Type I) involves
maximizing over both the hyperparameters and coefficients, while an empirical
Bayesian alternative (Type II) first marginalizes the coefficients and then
maximizes over the hyperparameters, leading to a tractable posterior
approximation. The underlying cost functions can be related via a dual-space
framework from Wipf et al. (2011), which allows both the Type I or Type II
objectives to be expressed in either coefficient or hyperparmeter space. This
perspective is useful because some analyses or extensions are more conducive to
development in one space or the other. Herein we consider the estimation of a
trade-off parameter balancing sparsity and data fit. As this parameter is
effectively a variance, natural estimators exist by assessing the problem in
hyperparameter (variance) space, transitioning natural ideas from Type II to
solve what is much less intuitive for Type I. In contrast, for analyses of
update rules and sparsity properties of local and global solutions, as well as
extensions to more general likelihood models, we can leverage coefficient-space
techniques developed for Type I and apply them to Type II. For example, this
allows us to prove that Type II-inspired techniques can be successful
recovering sparse coefficients when unfavorable restricted isometry properties
(RIP) lead to failure of popular L1 reconstructions. It also facilitates the
analysis of Type II when non-Gaussian likelihood models lead to intractable
integrations.Comment: 9 pages, 2 figures, submission to NIPS 201
Analyzing sparse dictionaries for online learning with kernels
Many signal processing and machine learning methods share essentially the
same linear-in-the-parameter model, with as many parameters as available
samples as in kernel-based machines. Sparse approximation is essential in many
disciplines, with new challenges emerging in online learning with kernels. To
this end, several sparsity measures have been proposed in the literature to
quantify sparse dictionaries and constructing relevant ones, the most prolific
ones being the distance, the approximation, the coherence and the Babel
measures. In this paper, we analyze sparse dictionaries based on these
measures. By conducting an eigenvalue analysis, we show that these sparsity
measures share many properties, including the linear independence condition and
inducing a well-posed optimization problem. Furthermore, we prove that there
exists a quasi-isometry between the parameter (i.e., dual) space and the
dictionary's induced feature space.Comment: 10 page
A dual framework for low-rank tensor completion
One of the popular approaches for low-rank tensor completion is to use the
latent trace norm regularization. However, most existing works in this
direction learn a sparse combination of tensors. In this work, we fill this gap
by proposing a variant of the latent trace norm that helps in learning a
non-sparse combination of tensors. We develop a dual framework for solving the
low-rank tensor completion problem. We first show a novel characterization of
the dual solution space with an interesting factorization of the optimal
solution. Overall, the optimal solution is shown to lie on a Cartesian product
of Riemannian manifolds. Furthermore, we exploit the versatile Riemannian
optimization framework for proposing computationally efficient trust region
algorithm. The experiments illustrate the efficacy of the proposed algorithm on
several real-world datasets across applications.Comment: Aceepted to appear in Advances of Nueral Information Processing
Systems (NIPS), 2018. A shorter version appeared in the NIPS workshop on
Synergies in Geometric Data Analysis 201
A General Framework of Dual Certificate Analysis for Structured Sparse Recovery Problems
This paper develops a general theoretical framework to analyze structured
sparse recovery problems using the notation of dual certificate. Although
certain aspects of the dual certificate idea have already been used in some
previous work, due to the lack of a general and coherent theory, the analysis
has so far only been carried out in limited scopes for specific problems. In
this context the current paper makes two contributions. First, we introduce a
general definition of dual certificate, which we then use to develop a unified
theory of sparse recovery analysis for convex programming. Second, we present a
class of structured sparsity regularization called structured Lasso for which
calculations can be readily performed under our theoretical framework. This new
theory includes many seemingly loosely related previous work as special cases;
it also implies new results that improve existing ones even for standard
formulations such as L1 regularization
Rank-Sparsity Incoherence for Matrix Decomposition
Suppose we are given a matrix that is formed by adding an unknown sparse
matrix to an unknown low-rank matrix. Our goal is to decompose the given matrix
into its sparse and low-rank components. Such a problem arises in a number of
applications in model and system identification, and is NP-hard in general. In
this paper we consider a convex optimization formulation to splitting the
specified matrix into its components, by minimizing a linear combination of the
norm and the nuclear norm of the components. We develop a notion of
\emph{rank-sparsity incoherence}, expressed as an uncertainty principle between
the sparsity pattern of a matrix and its row and column spaces, and use it to
characterize both fundamental identifiability as well as (deterministic)
sufficient conditions for exact recovery. Our analysis is geometric in nature,
with the tangent spaces to the algebraic varieties of sparse and low-rank
matrices playing a prominent role. When the sparse and low-rank matrices are
drawn from certain natural random ensembles, we show that the sufficient
conditions for exact recovery are satisfied with high probability. We conclude
with simulation results on synthetic matrix decomposition problems
An improved multi-parametric programming algorithm for flux balance analysis of metabolic networks
Flux balance analysis has proven an effective tool for analyzing metabolic
networks. In flux balance analysis, reaction rates and optimal pathways are
ascertained by solving a linear program, in which the growth rate is maximized
subject to mass-balance constraints. A variety of cell functions in response to
environmental stimuli can be quantified using flux balance analysis by
parameterizing the linear program with respect to extracellular conditions.
However, for most large, genome-scale metabolic networks of practical interest,
the resulting parametric problem has multiple and highly degenerate optimal
solutions, which are computationally challenging to handle. An improved
multi-parametric programming algorithm based on active-set methods is
introduced in this paper to overcome these computational difficulties.
Degeneracy and multiplicity are handled, respectively, by introducing
generalized inverses and auxiliary objective functions into the formulation of
the optimality conditions. These improvements are especially effective for
metabolic networks because their stoichiometry matrices are generally sparse;
thus, fast and efficient algorithms from sparse linear algebra can be leveraged
to compute generalized inverses and null-space bases. We illustrate the
application of our algorithm to flux balance analysis of metabolic networks by
studying a reduced metabolic model of Corynebacterium glutamicum and a
genome-scale model of Escherichia coli. We then demonstrate how the critical
regions resulting from these studies can be associated with optimal metabolic
modes and discuss the physical relevance of optimal pathways arising from
various auxiliary objective functions. Achieving more than five-fold
improvement in computational speed over existing multi-parametric programming
tools, the proposed algorithm proves promising in handling genome-scale
metabolic models.Comment: Accepted in J. Optim. Theory Appl. First draft was submitted on
August 4th, 201
Graph Regularized Tensor Sparse Coding for Image Representation
Sparse coding (SC) is an unsupervised learning scheme that has received an
increasing amount of interests in recent years. However, conventional SC
vectorizes the input images, which destructs the intrinsic spatial structures
of the images. In this paper, we propose a novel graph regularized tensor
sparse coding (GTSC) for image representation. GTSC preserves the local
proximity of elementary structures in the image by adopting the newly proposed
tubal-tensor representation. Simultaneously, it considers the intrinsic
geometric properties by imposing graph regularization that has been
successfully applied to uncover the geometric distribution for the image data.
Moreover, the returned sparse representations by GTSC have better physical
explanations as the key operation (i.e., circular convolution) in the
tubal-tensor model preserves the shifting invariance property. Experimental
results on image clustering demonstrate the effectiveness of the proposed
scheme
- …