Algorithms for nonnegative matrix factorization with the beta-divergence

This paper describes algorithms for nonnegative matrix factorization (NMF) with the beta-divergence (beta-NMF). The beta-divergence is a family of cost functions parametrized by a single shape parameter beta that takes the Euclidean distance, the Kullback-Leibler divergence and the Itakura-Saito divergence as special cases (beta = 2,1,0, respectively). The proposed algorithms are based on a surrogate auxiliary function (a local majorization of the criterion function). We first describe a majorization-minimization (MM) algorithm that leads to multiplicative updates, which differ from standard heuristic multiplicative updates by a beta-dependent power exponent. The monotonicity of the heuristic algorithm can however be proven for beta in (0,1) using the proposed auxiliary function. Then we introduce the concept of majorization-equalization (ME) algorithm which produces updates that move along constant level sets of the auxiliary function and lead to larger steps than MM. Simulations on synthetic and real data illustrate the faster convergence of the ME approach. The paper also describes how the proposed algorithms can be adapted to two common variants of NMF : penalized NMF (i.e., when a penalty function of the factors is added to the criterion function) and convex-NMF (when the dictionary is assumed to belong to a known subspace).Comment: \`a para\^itre dans Neural Computatio

A review on initialization methods for nonnegative matrix factorization: Towards omics data experiments

Nonnegative Matrix Factorization (NMF) has acquired a relevant role in the panorama of knowledge extraction, thanks to the peculiarity that non-negativity applies to both bases and weights, which allows meaningful interpretations and is consistent with the natural human part-based learning process. Nevertheless, most NMF algorithms are iterative, so initialization methods affect convergence behaviour, the quality of the final solution, and NMF performance in terms of the residual of the cost function. Studies on the impact of NMF initialization techniques have been conducted for text or image datasets, but very few considerations can be found in the literature when biological datasets are studied, even though NMFs have largely demonstrated their usefulness in better understanding biological mechanisms with omic datasets. This paper aims to present the state-of-the-art on NMF initialization schemes along with some initial considerations on the impact of initialization methods when microarrays (a simple instance of omic data) are evaluated with NMF mechanisms. Using a series of measures to qualitatively examine the biological information extracted by a given NMF scheme, it preliminary appears that some information (e.g., represented by genes) can be extracted regardless of the initialization scheme used

Scalable and distributed constrained low rank approximations

Low rank approximation is the problem of finding two low rank factors W and H such that the rank(WH) << rank(A) and A ≈ WH. These low rank factors W and H can be constrained for meaningful physical interpretation and referred as Constrained Low Rank Approximation (CLRA). Like most of the constrained optimization problem, performing CLRA can be computationally expensive than its unconstrained counterpart. A widely used CLRA is the Non-negative Matrix Factorization (NMF) which enforces non-negativity constraints in each of its low rank factors W and H. In this thesis, I focus on scalable/distributed CLRA algorithms for constraints such as boundedness and non-negativity for large real world matrices that includes text, High Definition (HD) video, social networks and recommender systems. First, I begin with the Bounded Matrix Low Rank Approximation (BMA) which imposes a lower and an upper bound on every element of the lower rank matrix. BMA is more challenging than NMF as it imposes bounds on the product WH rather than on each of the low rank factors W and H. For very large input matrices, we extend our BMA algorithm to Block BMA that can scale to a large number of processors. In applications, such as HD video, where the input matrix to be factored is extremely large, distributed computation is inevitable and the network communication becomes a major performance bottleneck. Towards this end, we propose a novel distributed Communication Avoiding NMF (CANMF) algorithm that communicates only the right low rank factor to its neighboring machine. Finally, a general distributed HPC- NMF framework that uses HPC techniques in communication intensive NMF operations and suitable for broader class of NMF algorithms.Ph.D

Evolutionary nonnegative matrix factorization with adaptive control of cluster quality

Nonnegative matrix factorization (NMF) approximates a given data matrix using linear combinations of a small number of nonnegative basis vectors, weighted by nonnegative encoding coefficients. This enables the exploration of the cluster structure of the data through the examination of the values of the encoding coefficients and therefore, NMF is often used as a popular tool for clustering analysis. However, its encoding coefficients do not always reveal a satisfactory cluster structure. To improve its effectiveness, a novel evolutionary strategy is proposed here to drive the iterative updating scheme of NMF and generate encoding coefficients of higher quality that are capable of offering more accurate and sharper cluster structures. The proposed hybridization procedure that relies on multiple initializations reinforces the robustness of the solution. Additionally, three evolving rules are designed to simultaneously boost the cluster quality and the reconstruction error during the iterative updates. Any clustering performance measure, such as either an internal one relying on the data itself or an external based on the availability of ground truth information, can be employed to drive the evolving procedure. The effectiveness of the proposed method is demonstrated via careful experimental designs and thorough comparative analyses using multiple benchmark datasets

Subspace Structure Regularized Nonnegative Matrix Factorization for Hyperspectral Unmixing

Hyperspectral unmixing is a crucial task for hyperspectral images (HSI) processing, which estimates the proportions of constituent materials of a mixed pixel. Usually, the mixed pixels can be approximated using a linear mixing model. Since each material only occurs in a few pixels in real HSI, sparse nonnegative matrix factorization (NMF) and its extensions are widely used as solutions. Some recent works assume that materials are distributed in certain structures, which can be added as constraints to sparse NMF model. However, they only consider the spatial distribution within a local neighborhood and define the distribution structure manually, while ignoring the real distribution of materials that is diverse in different images. In this paper, we propose a new unmixing method that learns a subspace structure from the original image and incorporate it into the sparse NMF framework to promote unmixing performance. Based on the self-representation property of data points lying in the same subspace, the learned subspace structure can indicate the global similar graph of pixels that represents the real distribution of materials. Then the similar graph is used as a robust global spatial prior which is expected to be maintained in the decomposed abundance matrix. The experiments conducted on both simulated and real-world HSI datasets demonstrate the superior performance of our proposed method