748 research outputs found
A Unified Framework for Stochastic Matrix Factorization via Variance Reduction
We propose a unified framework to speed up the existing stochastic matrix
factorization (SMF) algorithms via variance reduction. Our framework is general
and it subsumes several well-known SMF formulations in the literature. We
perform a non-asymptotic convergence analysis of our framework and derive
computational and sample complexities for our algorithm to converge to an
-stationary point in expectation. In addition, extensive experiments
for a wide class of SMF formulations demonstrate that our framework
consistently yields faster convergence and a more accurate output dictionary
vis-\`a-vis state-of-the-art frameworks
Harnessing Structures in Big Data via Guaranteed Low-Rank Matrix Estimation
Low-rank modeling plays a pivotal role in signal processing and machine
learning, with applications ranging from collaborative filtering, video
surveillance, medical imaging, to dimensionality reduction and adaptive
filtering. Many modern high-dimensional data and interactions thereof can be
modeled as lying approximately in a low-dimensional subspace or manifold,
possibly with additional structures, and its proper exploitations lead to
significant reduction of costs in sensing, computation and storage. In recent
years, there is a plethora of progress in understanding how to exploit low-rank
structures using computationally efficient procedures in a provable manner,
including both convex and nonconvex approaches. On one side, convex relaxations
such as nuclear norm minimization often lead to statistically optimal
procedures for estimating low-rank matrices, where first-order methods are
developed to address the computational challenges; on the other side, there is
emerging evidence that properly designed nonconvex procedures, such as
projected gradient descent, often provide globally optimal solutions with a
much lower computational cost in many problems. This survey article will
provide a unified overview of these recent advances on low-rank matrix
estimation from incomplete measurements. Attention is paid to rigorous
characterization of the performance of these algorithms, and to problems where
the low-rank matrix have additional structural properties that require new
algorithmic designs and theoretical analysis.Comment: To appear in IEEE Signal Processing Magazin
Low-Rank Modeling and Its Applications in Image Analysis
Low-rank modeling generally refers to a class of methods that solve problems
by representing variables of interest as low-rank matrices. It has achieved
great success in various fields including computer vision, data mining, signal
processing and bioinformatics. Recently, much progress has been made in
theories, algorithms and applications of low-rank modeling, such as exact
low-rank matrix recovery via convex programming and matrix completion applied
to collaborative filtering. These advances have brought more and more
attentions to this topic. In this paper, we review the recent advance of
low-rank modeling, the state-of-the-art algorithms, and related applications in
image analysis. We first give an overview to the concept of low-rank modeling
and challenging problems in this area. Then, we summarize the models and
algorithms for low-rank matrix recovery and illustrate their advantages and
limitations with numerical experiments. Next, we introduce a few applications
of low-rank modeling in the context of image analysis. Finally, we conclude
this paper with some discussions.Comment: To appear in ACM Computing Survey
Nonconvex and Nonsmooth Sparse Optimization via Adaptively Iterative Reweighted Methods
We present a general formulation of nonconvex and nonsmooth sparse
optimization problems with a convexset constraint, which takes into account
most existing types of nonconvex sparsity-inducing terms. It thus brings strong
applicability to a wide range of applications. We further design a general
algorithmic framework of adaptively iterative reweighted algorithms for solving
the nonconvex and nonsmooth sparse optimization problems. This is achieved by
solving a sequence of weighted convex penalty subproblems with adaptively
updated weights. The first-order optimality condition is then derived and the
global convergence results are provided under loose assumptions. This makes our
theoretical results a practical tool for analyzing a family of various
iteratively reweighted algorithms. In particular, for the iteratively reweighed
-algorithm, global convergence analysis is provided for cases with
diminishing relaxation parameter. For the iteratively reweighed
-algorithm, adaptively decreasing relaxation parameter is applicable
and the existence of the cluster point to the algorithm is established. The
effectiveness and efficiency of our proposed formulation and the algorithms are
demonstrated in numerical experiments in various sparse optimization problems
Model-free Nonconvex Matrix Completion: Local Minima Analysis and Applications in Memory-efficient Kernel PCA
This work studies low-rank approximation of a positive semidefinite matrix
from partial entries via nonconvex optimization. We characterized how well
local-minimum based low-rank factorization approximates a fixed positive
semidefinite matrix without any assumptions on the rank-matching, the condition
number or eigenspace incoherence parameter. Furthermore, under certain
assumptions on rank-matching and well-boundedness of condition numbers and
eigenspace incoherence parameters, a corollary of our main theorem improves the
state-of-the-art sampling rate results for nonconvex matrix completion with no
spurious local minima in Ge et al. [2016, 2017]. In addition, we investigated
when the proposed nonconvex optimization results in accurate low-rank
approximations even in presence of large condition numbers, large incoherence
parameters, or rank mismatching. We also propose to apply the nonconvex
optimization to memory-efficient Kernel PCA. Compared to the well-known
Nystr\"{o}m methods, numerical experiments indicate that the proposed nonconvex
optimization approach yields more stable results in both low-rank approximation
and clustering.Comment: Main theorem improve
The proximal point method revisited
In this short survey, I revisit the role of the proximal point method in
large scale optimization. I focus on three recent examples: a proximally guided
subgradient method for weakly convex stochastic approximation, the prox-linear
algorithm for minimizing compositions of convex functions and smooth maps, and
Catalyst generic acceleration for regularized Empirical Risk Minimization.Comment: 11 pages, submitted to SIAG/OPT Views and New
Global Optimality in Low-rank Matrix Optimization
This paper considers the minimization of a general objective function
over the set of rectangular matrices that have rank at most . To
reduce the computational burden, we factorize the variable into a product
of two smaller matrices and optimize over these two matrices instead of .
Despite the resulting nonconvexity, recent studies in matrix completion and
sensing have shown that the factored problem has no spurious local minima and
obeys the so-called strict saddle property (the function has a directional
negative curvature at all critical points but local minima). We analyze the
global geometry for a general and yet well-conditioned objective function
whose restricted strong convexity and restricted strong smoothness
constants are comparable. In particular, we show that the reformulated
objective function has no spurious local minima and obeys the strict saddle
property. These geometric properties imply that a number of iterative
optimization algorithms (such as gradient descent) can provably solve the
factored problem with global convergence
Nonconvex Sparse Learning via Stochastic Optimization with Progressive Variance Reduction
We propose a stochastic variance reduced optimization algorithm for solving
sparse learning problems with cardinality constraints. Sufficient conditions
are provided, under which the proposed algorithm enjoys strong linear
convergence guarantees and optimal estimation accuracy in high dimensions. We
further extend the proposed algorithm to an asynchronous parallel variant with
a near linear speedup. Numerical experiments demonstrate the efficiency of our
algorithm in terms of both parameter estimation and computational performance
Decomposition into Low-rank plus Additive Matrices for Background/Foreground Separation: A Review for a Comparative Evaluation with a Large-Scale Dataset
Recent research on problem formulations based on decomposition into low-rank
plus sparse matrices shows a suitable framework to separate moving objects from
the background. The most representative problem formulation is the Robust
Principal Component Analysis (RPCA) solved via Principal Component Pursuit
(PCP) which decomposes a data matrix in a low-rank matrix and a sparse matrix.
However, similar robust implicit or explicit decompositions can be made in the
following problem formulations: Robust Non-negative Matrix Factorization
(RNMF), Robust Matrix Completion (RMC), Robust Subspace Recovery (RSR), Robust
Subspace Tracking (RST) and Robust Low-Rank Minimization (RLRM). The main goal
of these similar problem formulations is to obtain explicitly or implicitly a
decomposition into low-rank matrix plus additive matrices. In this context,
this work aims to initiate a rigorous and comprehensive review of the similar
problem formulations in robust subspace learning and tracking based on
decomposition into low-rank plus additive matrices for testing and ranking
existing algorithms for background/foreground separation. For this, we first
provide a preliminary review of the recent developments in the different
problem formulations which allows us to define a unified view that we called
Decomposition into Low-rank plus Additive Matrices (DLAM). Then, we examine
carefully each method in each robust subspace learning/tracking frameworks with
their decomposition, their loss functions, their optimization problem and their
solvers. Furthermore, we investigate if incremental algorithms and real-time
implementations can be achieved for background/foreground separation. Finally,
experimental results on a large-scale dataset called Background Models
Challenge (BMC 2012) show the comparative performance of 32 different robust
subspace learning/tracking methods.Comment: 121 pages, 5 figures, submitted to Computer Science Review. arXiv
admin note: text overlap with arXiv:1312.7167, arXiv:1109.6297,
arXiv:1207.3438, arXiv:1105.2126, arXiv:1404.7592, arXiv:1210.0805,
arXiv:1403.8067 by other authors, Computer Science Review, November 201
Simple and practical algorithms for -norm low-rank approximation
We propose practical algorithms for entrywise -norm low-rank
approximation, for or . The proposed framework, which is
non-convex and gradient-based, is easy to implement and typically attains
better approximations, faster, than state of the art.
From a theoretical standpoint, we show that the proposed scheme can attain
-OPT approximations. Our algorithms are not
hyperparameter-free: they achieve the desiderata only assuming algorithm's
hyperparameters are known a priori---or are at least approximable. I.e., our
theory indicates what problem quantities need to be known, in order to get a
good solution within polynomial time, and does not contradict to recent
inapproximabilty results, as in [46].Comment: 16 pages, 11 figures, to appear in UAI 201
- …