1,530 research outputs found
Heuristics for Exact Nonnegative Matrix Factorization
The exact nonnegative matrix factorization (exact NMF) problem is the
following: given an -by- nonnegative matrix and a factorization rank
, find, if possible, an -by- nonnegative matrix and an -by-
nonnegative matrix such that . In this paper, we propose two
heuristics for exact NMF, one inspired from simulated annealing and the other
from the greedy randomized adaptive search procedure. We show that these two
heuristics are able to compute exact nonnegative factorizations for several
classes of nonnegative matrices (namely, linear Euclidean distance matrices,
slack matrices, unique-disjointness matrices, and randomly generated matrices)
and as such demonstrate their superiority over standard multi-start strategies.
We also consider a hybridization between these two heuristics that allows us to
combine the advantages of both methods. Finally, we discuss the use of these
heuristics to gain insight on the behavior of the nonnegative rank, i.e., the
minimum factorization rank such that an exact NMF exists. In particular, we
disprove a conjecture on the nonnegative rank of a Kronecker product, propose a
new upper bound on the extension complexity of generic -gons and conjecture
the exact value of (i) the extension complexity of regular -gons and (ii)
the nonnegative rank of a submatrix of the slack matrix of the correlation
polytope.Comment: 32 pages, 2 figures, 16 table
Computing a Nonnegative Matrix Factorization -- Provably
In the Nonnegative Matrix Factorization (NMF) problem we are given an nonnegative matrix and an integer . Our goal is to express
as where and are nonnegative matrices of size
and respectively. In some applications, it makes sense to ask
instead for the product to approximate -- i.e. (approximately)
minimize \norm{M - AW}_F where \norm{}_F denotes the Frobenius norm; we
refer to this as Approximate NMF. This problem has a rich history spanning
quantum mechanics, probability theory, data analysis, polyhedral combinatorics,
communication complexity, demography, chemometrics, etc. In the past decade NMF
has become enormously popular in machine learning, where and are
computed using a variety of local search heuristics. Vavasis proved that this
problem is NP-complete. We initiate a study of when this problem is solvable in
polynomial time:
1. We give a polynomial-time algorithm for exact and approximate NMF for
every constant . Indeed NMF is most interesting in applications precisely
when is small.
2. We complement this with a hardness result, that if exact NMF can be solved
in time , 3-SAT has a sub-exponential time algorithm. This rules
out substantial improvements to the above algorithm.
3. We give an algorithm that runs in time polynomial in , and
under the separablity condition identified by Donoho and Stodden in 2003. The
algorithm may be practical since it is simple and noise tolerant (under benign
assumptions). Separability is believed to hold in many practical settings.
To the best of our knowledge, this last result is the first example of a
polynomial-time algorithm that provably works under a non-trivial condition on
the input and we believe that this will be an interesting and important
direction for future work.Comment: 29 pages, 3 figure
Learning Topic Models - Going beyond SVD
Topic Modeling is an approach used for automatic comprehension and
classification of data in a variety of settings, and perhaps the canonical
application is in uncovering thematic structure in a corpus of documents. A
number of foundational works both in machine learning and in theory have
suggested a probabilistic model for documents, whereby documents arise as a
convex combination of (i.e. distribution on) a small number of topic vectors,
each topic vector being a distribution on words (i.e. a vector of
word-frequencies). Similar models have since been used in a variety of
application areas; the Latent Dirichlet Allocation or LDA model of Blei et al.
is especially popular.
Theoretical studies of topic modeling focus on learning the model's
parameters assuming the data is actually generated from it. Existing approaches
for the most part rely on Singular Value Decomposition(SVD), and consequently
have one of two limitations: these works need to either assume that each
document contains only one topic, or else can only recover the span of the
topic vectors instead of the topic vectors themselves.
This paper formally justifies Nonnegative Matrix Factorization(NMF) as a main
tool in this context, which is an analog of SVD where all vectors are
nonnegative. Using this tool we give the first polynomial-time algorithm for
learning topic models without the above two limitations. The algorithm uses a
fairly mild assumption about the underlying topic matrix called separability,
which is usually found to hold in real-life data. A compelling feature of our
algorithm is that it generalizes to models that incorporate topic-topic
correlations, such as the Correlated Topic Model and the Pachinko Allocation
Model.
We hope that this paper will motivate further theoretical results that use
NMF as a replacement for SVD - just as NMF has come to replace SVD in many
applications
Factoring nonnegative matrices with linear programs
This paper describes a new approach, based on linear programming, for
computing nonnegative matrix factorizations (NMFs). The key idea is a
data-driven model for the factorization where the most salient features in the
data are used to express the remaining features. More precisely, given a data
matrix X, the algorithm identifies a matrix C such that X approximately equals
CX and some linear constraints. The constraints are chosen to ensure that the
matrix C selects features; these features can then be used to find a low-rank
NMF of X. A theoretical analysis demonstrates that this approach has guarantees
similar to those of the recent NMF algorithm of Arora et al. (2012). In
contrast with this earlier work, the proposed method extends to more general
noise models and leads to efficient, scalable algorithms. Experiments with
synthetic and real datasets provide evidence that the new approach is also
superior in practice. An optimized C++ implementation can factor a
multigigabyte matrix in a matter of minutes.Comment: 17 pages, 10 figures. Modified theorem statement for robust recovery
conditions. Revised proof techniques to make arguments more elementary.
Results on robustness when rows are duplicated have been superseded by
arxiv.org/1211.668
A geometric approach to archetypal analysis and non-negative matrix factorization
Archetypal analysis and non-negative matrix factorization (NMF) are staples
in a statisticians toolbox for dimension reduction and exploratory data
analysis. We describe a geometric approach to both NMF and archetypal analysis
by interpreting both problems as finding extreme points of the data cloud. We
also develop and analyze an efficient approach to finding extreme points in
high dimensions. For modern massive datasets that are too large to fit on a
single machine and must be stored in a distributed setting, our approach makes
only a small number of passes over the data. In fact, it is possible to obtain
the NMF or perform archetypal analysis with just two passes over the data.Comment: 36 pages, 13 figure
Algorithms for Positive Semidefinite Factorization
This paper considers the problem of positive semidefinite factorization (PSD
factorization), a generalization of exact nonnegative matrix factorization.
Given an -by- nonnegative matrix and an integer , the PSD
factorization problem consists in finding, if possible, symmetric -by-
positive semidefinite matrices and such
that for , and . PSD
factorization is NP-hard. In this work, we introduce several local optimization
schemes to tackle this problem: a fast projected gradient method and two
algorithms based on the coordinate descent framework. The main application of
PSD factorization is the computation of semidefinite extensions, that is, the
representations of polyhedrons as projections of spectrahedra, for which the
matrix to be factorized is the slack matrix of the polyhedron. We compare the
performance of our algorithms on this class of problems. In particular, we
compute the PSD extensions of size for the
regular -gons when , and . We also show how to generalize our
algorithms to compute the square root rank (which is the size of the factors in
a PSD factorization where all factor matrices and have rank one)
and completely PSD factorizations (which is the special case where the input
matrix is symmetric and equality is required for all ).Comment: 21 pages, 3 figures, 3 table
- …