193 research outputs found
The Diagonalized Newton Algorithm for Nonnegative Matrix Factorization
Non-negative matrix factorization (NMF) has become a popular machine learning
approach to many problems in text mining, speech and image processing,
bio-informatics and seismic data analysis to name a few. In NMF, a matrix of
non-negative data is approximated by the low-rank product of two matrices with
non-negative entries. In this paper, the approximation quality is measured by
the Kullback-Leibler divergence between the data and its low-rank
reconstruction. The existence of the simple multiplicative update (MU)
algorithm for computing the matrix factors has contributed to the success of
NMF. Despite the availability of algorithms showing faster convergence, MU
remains popular due to its simplicity. In this paper, a diagonalized Newton
algorithm (DNA) is proposed showing faster convergence while the implementation
remains simple and suitable for high-rank problems. The DNA algorithm is
applied to various publicly available data sets, showing a substantial speed-up
on modern hardware.Comment: 8 pages + references; International Conference on Learning
Representations, 201
Convergence of Gradient Descent for Low-Rank Matrix Approximation
This paper provides a proof of global convergence of gradient search for low-rank matrix approximation. Such approximations have recently been of interest for large-scale problems, as well as for dictionary learning for sparse signal representations and matrix completion. The proof is based on the interpretation of the problem as an optimization on the Grassmann manifold and Fubiny-Study distance on this space
Computing Large-Scale Matrix and Tensor Decomposition with Structured Factors: A Unified Nonconvex Optimization Perspective
The proposed article aims at offering a comprehensive tutorial for the
computational aspects of structured matrix and tensor factorization. Unlike
existing tutorials that mainly focus on {\it algorithmic procedures} for a
small set of problems, e.g., nonnegativity or sparsity-constrained
factorization, we take a {\it top-down} approach: we start with general
optimization theory (e.g., inexact and accelerated block coordinate descent,
stochastic optimization, and Gauss-Newton methods) that covers a wide range of
factorization problems with diverse constraints and regularization terms of
engineering interest. Then, we go `under the hood' to showcase specific
algorithm design under these introduced principles. We pay a particular
attention to recent algorithmic developments in structured tensor and matrix
factorization (e.g., random sketching and adaptive step size based stochastic
optimization and structure-exploiting second-order algorithms), which are the
state of the art---yet much less touched upon in the literature compared to
{\it block coordinate descent} (BCD)-based methods. We expect that the article
to have an educational values in the field of structured factorization and hope
to stimulate more research in this important and exciting direction.Comment: Final Version; to appear in IEEE Signal Processing Magazine; title
revised to comply with the journal's rul
Accurate and Efficient Expression Evaluation and Linear Algebra
We survey and unify recent results on the existence of accurate algorithms
for evaluating multivariate polynomials, and more generally for accurate
numerical linear algebra with structured matrices. By "accurate" we mean that
the computed answer has relative error less than 1, i.e., has some correct
leading digits. We also address efficiency, by which we mean algorithms that
run in polynomial time in the size of the input. Our results will depend
strongly on the model of arithmetic: Most of our results will use the so-called
Traditional Model (TM). We give a set of necessary and sufficient conditions to
decide whether a high accuracy algorithm exists in the TM, and describe
progress toward a decision procedure that will take any problem and provide
either a high accuracy algorithm or a proof that none exists. When no accurate
algorithm exists in the TM, it is natural to extend the set of available
accurate operations by a library of additional operations, such as , dot
products, or indeed any enumerable set which could then be used to build
further accurate algorithms. We show how our accurate algorithms and decision
procedure for finding them extend to this case. Finally, we address other
models of arithmetic, and the relationship between (im)possibility in the TM
and (in)efficient algorithms operating on numbers represented as bit strings.Comment: 49 pages, 6 figures, 1 tabl
Regularized Compression of A Noisy Blurred Image
Both regularization and compression are important issues in image processing and have been widely
approached in the literature. The usual procedure to obtain the compression of an image given through a
noisy blur requires two steps: first a deblurring step of the image and then a factorization step of the
regularized image to get an approximation in terms of low rank nonnegative factors. We examine here the
possibility of swapping the two steps by deblurring directly the noisy factors or partially denoised factors.
The experimentation shows that in this way images with comparable regularized compression can be
obtained with a lower computational cost
Expansions and factorizations of matrices and their applications
Abstract. Linear algebra is a foundation to decompositions and algorithms for extracting simple structures from complex data. In this thesis, we investigate and apply modern techniques from linear algebra to solve problems arising in signal processing and computer science. In particular, we focus on data that takes the shape of a matrix and we explore how to represent it as products of circulant and diagonal matrices. To this end, we study matrix decompositions, approximations, and structured matrix expansions whose elements are products of circulant and diagonal matrices. Computationally, we develop a matrix expansion with DCD matrices for approximating a given matrix. Remarkably, DCD matrices, i.e., a product of diagonal matrix, circulant matrix, and another diagonal matrix, give an natural extension to rank-one matrices. Inspired from the singular value decomposition, we introduce a notion of a matrix rank closely related to the expansion and compute the rank of some specific structured matrices. Specifically, Toeplitz matrix is a sum of two DCD matrices. Here, we present a greedy algorithmic framework to devise the expansion numerically. Finally, we show that the practical uses of the DCD expansion can be complemented by the proposed framework and perform two experiments with a view towards applications
- …