141 research outputs found
Overcomplete Independent Component Analysis via SDP
We present a novel algorithm for overcomplete independent components analysis
(ICA), where the number of latent sources k exceeds the dimension p of observed
variables. Previous algorithms either suffer from high computational complexity
or make strong assumptions about the form of the mixing matrix. Our algorithm
does not make any sparsity assumption yet enjoys favorable computational and
theoretical properties. Our algorithm consists of two main steps: (a)
estimation of the Hessians of the cumulant generating function (as opposed to
the fourth and higher order cumulants used by most algorithms) and (b) a novel
semi-definite programming (SDP) relaxation for recovering a mixing component.
We show that this relaxation can be efficiently solved with a projected
accelerated gradient descent method, which makes the whole algorithm
computationally practical. Moreover, we conjecture that the proposed program
recovers a mixing component at the rate k < p^2/4 and prove that a mixing
component can be recovered with high probability when k < (2 - epsilon) p log p
when the original components are sampled uniformly at random on the hyper
sphere. Experiments are provided on synthetic data and the CIFAR-10 dataset of
real images.Comment: Appears in: Proceedings of the 22nd International Conference on
Artificial Intelligence and Statistics (AISTATS 2019). 21 page
Decomposing Overcomplete 3rd Order Tensors using Sum-of-Squares Algorithms
Tensor rank and low-rank tensor decompositions have many applications in
learning and complexity theory. Most known algorithms use unfoldings of tensors
and can only handle rank up to for a -th order
tensor in . Previously no efficient algorithm can decompose
3rd order tensors when the rank is super-linear in the dimension. Using ideas
from sum-of-squares hierarchy, we give the first quasi-polynomial time
algorithm that can decompose a random 3rd order tensor decomposition when the
rank is as large as .
We also give a polynomial time algorithm for certifying the injective norm of
random low rank tensors. Our tensor decomposition algorithm exploits the
relationship between injective norm and the tensor components. The proof relies
on interesting tools for decoupling random variables to prove better matrix
concentration bounds, which can be useful in other settings
Sum-of-squares proofs and the quest toward optimal algorithms
In order to obtain the best-known guarantees, algorithms are traditionally
tailored to the particular problem we want to solve. Two recent developments,
the Unique Games Conjecture (UGC) and the Sum-of-Squares (SOS) method,
surprisingly suggest that this tailoring is not necessary and that a single
efficient algorithm could achieve best possible guarantees for a wide range of
different problems.
The Unique Games Conjecture (UGC) is a tantalizing conjecture in
computational complexity, which, if true, will shed light on the complexity of
a great many problems. In particular this conjecture predicts that a single
concrete algorithm provides optimal guarantees among all efficient algorithms
for a large class of computational problems.
The Sum-of-Squares (SOS) method is a general approach for solving systems of
polynomial constraints. This approach is studied in several scientific
disciplines, including real algebraic geometry, proof complexity, control
theory, and mathematical programming, and has found applications in fields as
diverse as quantum information theory, formal verification, game theory and
many others.
We survey some connections that were recently uncovered between the Unique
Games Conjecture and the Sum-of-Squares method. In particular, we discuss new
tools to rigorously bound the running time of the SOS method for obtaining
approximate solutions to hard optimization problems, and how these tools give
the potential for the sum-of-squares method to provide new guarantees for many
problems of interest, and possibly to even refute the UGC.Comment: Survey. To appear in proceedings of ICM 201
Rank-Sparsity Incoherence for Matrix Decomposition
Suppose we are given a matrix that is formed by adding an unknown sparse
matrix to an unknown low-rank matrix. Our goal is to decompose the given matrix
into its sparse and low-rank components. Such a problem arises in a number of
applications in model and system identification, and is NP-hard in general. In
this paper we consider a convex optimization formulation to splitting the
specified matrix into its components, by minimizing a linear combination of the
norm and the nuclear norm of the components. We develop a notion of
\emph{rank-sparsity incoherence}, expressed as an uncertainty principle between
the sparsity pattern of a matrix and its row and column spaces, and use it to
characterize both fundamental identifiability as well as (deterministic)
sufficient conditions for exact recovery. Our analysis is geometric in nature,
with the tangent spaces to the algebraic varieties of sparse and low-rank
matrices playing a prominent role. When the sparse and low-rank matrices are
drawn from certain natural random ensembles, we show that the sufficient
conditions for exact recovery are satisfied with high probability. We conclude
with simulation results on synthetic matrix decomposition problems
- …