Sufficient conditions for convergence of the Sum-Product Algorithm

We derive novel conditions that guarantee convergence of the Sum-Product algorithm (also known as Loopy Belief Propagation or simply Belief Propagation) to a unique fixed point, irrespective of the initial messages. The computational complexity of the conditions is polynomial in the number of variables. In contrast with previously existing conditions, our results are directly applicable to arbitrary factor graphs (with discrete variables) and are shown to be valid also in the case of factors containing zeros, under some additional conditions. We compare our bounds with existing ones, numerically and, if possible, analytically. For binary variables with pairwise interactions, we derive sufficient conditions that take into account local evidence (i.e., single variable factors) and the type of pair interactions (attractive or repulsive). It is shown empirically that this bound outperforms existing bounds.Comment: 15 pages, 5 figures. Major changes and new results in this revised version. Submitted to IEEE Transactions on Information Theor

Inertial Douglas-Rachford splitting for monotone inclusion problems

We propose an inertial Douglas-Rachford splitting algorithm for finding the set of zeros of the sum of two maximally monotone operators in Hilbert spaces and investigate its convergence properties. To this end we formulate first the inertial version of the Krasnosel'ski\u{\i}--Mann algorithm for approximating the set of fixed points of a nonexpansive operator, for which we also provide an exhaustive convergence analysis. By using a product space approach we employ these results to the solving of monotone inclusion problems involving linearly composed and parallel-sum type operators and provide in this way iterative schemes where each of the maximally monotone mappings is accessed separately via its resolvent. We consider also the special instance of solving a primal-dual pair of nonsmooth convex optimization problems and illustrate the theoretical results via some numerical experiments in clustering and location theory.Comment: arXiv admin note: text overlap with arXiv:1402.529

Orthogonal Trace-Sum Maximization: Applications, Local Algorithms, and Global Optimality

This paper studies a problem of maximizing the sum of traces of matrix quadratic forms on a product of Stiefel manifolds. This orthogonal trace-sum maximization (OTSM) problem generalizes many interesting problems such as generalized canonical correlation analysis (CCA), Procrustes analysis, and cryo-electron microscopy of the Nobel prize fame. For these applications finding global solutions is highly desirable but has been out of reach for a long time. For example, generalizations of CCA do not possess obvious global solutions unlike their classical counterpart to which a global solution is readily obtained through singular value decomposition; it is also not clear how to test global optimality. We provide a simple method to certify global optimality of a given local solution. This method only requires testing the sign of the smallest eigenvalue of a symmetric matrix, and does not rely on a particular algorithm as long as it converges to a stationary point. Our certificate result relies on a semidefinite programming (SDP) relaxation of OTSM, but avoids solving an SDP of lifted dimensions. Surprisingly, a popular algorithm for generalized CCA and Procrustes analysis may generate oscillating iterates. We propose a simple modification of this standard algorithm and prove that it reliably converges. Our notion of convergence is stronger than conventional objective value convergence or subsequence convergence.The convergence result utilizes the Kurdyka-Lojasiewicz property of the problem.Comment: 22 pages, 1 figur

A Novel Partitioning Method for Accelerating the Block Cimmino Algorithm

We propose a novel block-row partitioning method in order to improve the convergence rate of the block Cimmino algorithm for solving general sparse linear systems of equations. The convergence rate of the block Cimmino algorithm depends on the orthogonality among the block rows obtained by the partitioning method. The proposed method takes numerical orthogonality among block rows into account by proposing a row inner-product graph model of the coefficient matrix. In the graph partitioning formulation defined on this graph model, the partitioning objective of minimizing the cutsize directly corresponds to minimizing the sum of inter-block inner products between block rows thus leading to an improvement in the eigenvalue spectrum of the iteration matrix. This in turn leads to a significant reduction in the number of iterations required for convergence. Extensive experiments conducted on a large set of matrices confirm the validity of the proposed method against a state-of-the-art method