Relative Pairwise Relationship Constrained Non-negative Matrix Factorisation

Non-negative Matrix Factorisation (NMF) has been extensively used in machine learning and data analytics applications. Most existing variations of NMF only consider how each row/column vector of factorised matrices should be shaped, and ignore the relationship among pairwise rows or columns. In many cases, such pairwise relationship enables better factorisation, for example, image clustering and recommender systems. In this paper, we propose an algorithm named, Relative Pairwise Relationship constrained Non-negative Matrix Factorisation (RPR-NMF), which places constraints over relative pairwise distances amongst features by imposing penalties in a triplet form. Two distance measures, squared Euclidean distance and Symmetric divergence, are used, and exponential and hinge loss penalties are adopted for the two measures respectively. It is well known that the so-called "multiplicative update rules" result in a much faster convergence than gradient descend for matrix factorisation. However, applying such update rules to RPR-NMF and also proving its convergence is not straightforward. Thus, we use reasonable approximations to relax the complexity brought by the penalties, which are practically verified. Experiments on both synthetic datasets and real datasets demonstrate that our algorithms have advantages on gaining close approximation, satisfying a high proportion of expected constraints, and achieving superior performance compared with other algorithms.Comment: 13 pages, 10 figure

Constrained Low-Rank Matrix/Tensor Factorisation

University of Technology Sydney. Faculty of Engineering and Information Technology.Constrained low-rank matrix and tensor factorisation (MF/TF) have been widely used in machine learning and data analytics. Studies on the way of modelling constraints and the solution of optimisation task in general can provide theoretical supports for applications like image clustering, recommender systems and data compression. This thesis studies three algorithms of constrained low-rank MF/TF. Imposing constraints on each feature vector of factor matrices is a common practice in many constrained low-rank MF algorithms. However, in many real scenarios, the relationships among features can influence the factorisation results as well. In order to better characterise the relationships among features, a novel MF algorithm, Relative Pairwise Relationship Constrained Non-negative Matrix Factorisation, is proposed. It places soft constraints over relative pairwise distances amongst features as regularisations to retain expected relationships after factorisation. It conforms to the so-called \multiplicative update rules" and detailed convergence proofs are provided. Experiments on both synthetic and real datasets have verified that imposing such constraints can keep most expected relationships unchanged after factorisation. Directly adopted on tensor data, low-rank TF can effectively avoid the information loss caused by matricisation. The relationships among features of factor matrices in TF have practical meanings in many real scenarios. To describe such relative relationships in low-rank TF, this thesis proposes Relative Pairwise Relationship Constrained Non-negative Tensor Factorisation. It deals with both Candecomp/Parafac and Tucker decomposition schemes and both squared Euclidean distance and divergence measures. The utilisation of tensor factorisation matricisation equation simplifies the update rules and greatly improves the computation efficiency. Experiments have demonstrated that the proposed algorithm can achieve higher accuracy when adopted on tensor applications. There exists a problem of acquiring out-of-bounds and fluctuating values over predictions when applying low-rank MF on recommender systems. The commonly used solutions, truncation and imposing penalties, can cause the decrease in the number of effective predictions and affect the recommendation accuracy. This thesis creatively proposes Magnitude Bounded Matrix Factorisation to handle the above problem by imposing magnitude constraints for the first time. It first converts the original quadratically constrained quadratic programming task to an unconstrained one which is then solved by the well-known stochastic gradient descent. An acceleration approach for improving computation efficiency, an extracting method for magnitude constraints and a variant of MBMF for non-negative data are also introduced. Experiments have demonstrated that the algorithm is superior to existing bounding algorithms on both computing efficiency and recommendation performance

Relative Pairwise Relationship Constrained Non-negative Matrix Factorisation

IEEE Non-negative Matrix Factorisation (NMF) has been extensively used in machine learning and data analytics applications. Most existing variations of NMF only consider how each row/column vector of factorised matrices should be shaped, and ignore the relationship among pairwise rows or columns. In many cases, such pairwise relationship enables better factorisation, for example, image clustering and recommender systems. In this paper, we propose an algorithm named, Relative Pairwise Relationship constrained Non-negative Matrix Factorisation (RPR-NMF), which places constraints over relative pairwise distances amongst features by imposing penalties in a triplet form. Two distance measures, squared Euclidean distance and Symmetric divergence, are used, and exponential and hinge loss penalties are adopted for the two measures respectively. It is well known that the so-called "multiplicative update rules" result in a much faster convergence than gradient descend for matrix factorisation. However, applying such update rules to RPR-NMF and also proving its convergence is not straightforward. Thus, we use reasonable approximations to relax the complexity brought by the penalties, which are practically verified. Experiments on both synthetic datasets and real datasets demonstrate that our algorithms have advantages on gaining close approximation, satisfying a high proportion of expected constraints, and achieving superior performance compared with other algorithms