Approximating the Permanent with Fractional Belief Propagation

We discuss schemes for exact and approximate computations of permanents, and compare them with each other. Specifically, we analyze the Belief Propagation (BP) approach and its Fractional Belief Propagation (FBP) generalization for computing the permanent of a non-negative matrix. Known bounds and conjectures are verified in experiments, and some new theoretical relations, bounds and conjectures are proposed. The Fractional Free Energy (FFE) functional is parameterized by a scalar parameter $\gamma\in[-1;1]$, where $\gamma=-1$ corresponds to the BP limit and $\gamma=1$ corresponds to the exclusion principle (but ignoring perfect matching constraints) Mean-Field (MF) limit. FFE shows monotonicity and continuity with respect to $\gamma$. For every non-negative matrix, we define its special value $\gamma_*\in[-1;0]$ to be the $\gamma$ for which the minimum of the $\gamma$-parameterized FFE functional is equal to the permanent of the matrix, where the lower and upper bounds of the $\gamma$-interval corresponds to respective bounds for the permanent. Our experimental analysis suggests that the distribution of $\gamma_*$ varies for different ensembles but $\gamma_*$ always lies within the $[-1;-1/2]$ interval. Moreover, for all ensembles considered the behavior of $\gamma_*$ is highly distinctive, offering an emprirical practical guidance for estimating permanents of non-negative matrices via the FFE approach.Comment: 42 pages, 14 figure

Degree-MM Bethe and Sinkhorn Permanent Based Bounds on the Permanent of a Non-negative Matrix

The permanent of a non-negative square matrix can be well approximated by finding the minimum of the Bethe free energy functions associated with some suitably defined factor graph; the resulting approximation to the permanent is called the Bethe permanent. Vontobel gave a combinatorial characterization of the Bethe permanent via degree-$M$ Bethe permanents, which is based on degree-$M$ covers of the underlying factor graph. In this paper, we prove a degree-$M$-Bethe-permanent-based lower bound on the permanent of a non-negative matrix, which solves a conjecture proposed by Vontobel in [IEEE Trans. Inf. Theory, Mar. 2013]. We also prove a degree-$M$-Bethe-permanent-based upper bound on the permanent of a non-negative matrix. In the limit $M \to \infty$, these lower and upper bounds yield known Bethe-permanent-based lower and upper bounds on the permanent of a non-negative matrix. Moreover, we prove similar results for an approximation to the permanent known as the (scaled) Sinkhorn permanent.Comment: submitte

Approximating the Permanent with Fractional Belief Propagation

We discuss schemes for exact and approximate computations of permanents, and compare them with each other. Specifically, we analyze the belief propagation (BP) approach and its fractional belief propagation (FBP) generalization for computing the permanent of a non-negative matrix. Known bounds and Conjectures are verified in experiments, and some new theoretical relations, bounds and Conjectures are proposed. The fractional free energy (FFE) function is parameterized by a scalar parameter y ∈ [−1;1], where y = −1 corresponds to the BP limit and y = 1 corresponds to the exclusion principle (but ignoring perfect matching constraints) mean-field (MF) limit. FFE shows monotonicity and continuity with respect to g. For every non-negative matrix, we define its special value y∗ ∈ [−1;0] to be the g for which the minimum of the y-parameterized FFE function is equal to the permanent of the matrix, where the lower and upper bounds of the g-interval corresponds to respective bounds for the permanent. Our experimental analysis suggests that the distribution of y∗ varies for different ensembles but y∗ always lies within the [−1;−1/2] interval. Moreover, for all ensembles considered, the behavior of y∗ is highly distinctive, offering an empirical practical guidance for estimating permanents of non-negative matrices via the FFE approach.Los Alamos National Laboratory (Undergraduate Research Assistant Program)United States. National Nuclear Security Administration (Los Alamos National Laboratory Contract DE C52-06NA25396

Spherical and Hyperbolic Toric Topology-Based Codes On Graph Embedding for Ising MRF Models: Classical and Quantum Topology Machine Learning

The paper introduces the application of information geometry to describe the ground states of Ising models by utilizing parity-check matrices of cyclic and quasi-cyclic codes on toric and spherical topologies. The approach establishes a connection between machine learning and error-correcting coding. This proposed approach has implications for the development of new embedding methods based on trapping sets. Statistical physics and number geometry applied for optimize error-correcting codes, leading to these embedding and sparse factorization methods. The paper establishes a direct connection between DNN architecture and error-correcting coding by demonstrating how state-of-the-art architectures (ChordMixer, Mega, Mega-chunk, CDIL, ...) from the long-range arena can be equivalent to of block and convolutional LDPC codes (Cage-graph, Repeat Accumulate). QC codes correspond to certain types of chemical elements, with the carbon element being represented by the mixed automorphism Shu-Lin-Fossorier QC-LDPC code. The connections between Belief Propagation and the Permanent, Bethe-Permanent, Nishimori Temperature, and Bethe-Hessian Matrix are elaborated upon in detail. The Quantum Approximate Optimization Algorithm (QAOA) used in the Sherrington-Kirkpatrick Ising model can be seen as analogous to the back-propagation loss function landscape in training DNNs. This similarity creates a comparable problem with TS pseudo-codeword, resembling the belief propagation method. Additionally, the layer depth in QAOA correlates to the number of decoding belief propagation iterations in the Wiberg decoding tree. Overall, this work has the potential to advance multiple fields, from Information Theory, DNN architecture design (sparse and structured prior graph topology), efficient hardware design for Quantum and Classical DPU/TPU (graph, quantize and shift register architect.) to Materials Science and beyond.Comment: 71 pages, 42 Figures, 1 Table, 1 Appendix. arXiv admin note: text overlap with arXiv:2109.08184 by other author

Factor Graphs for Quantum Information Processing

[...] In this thesis, we are interested in generalizing factor graphs and the relevant methods toward describing quantum systems. Two generalizations of classical graphical models are investigated, namely double-edge factor graphs (DeFGs) and quantum factor graphs (QFGs). Conventionally, a factor in a factor graph represents a nonnegative real-valued local functions. Two different approaches to generalize factors in classical factor graphs yield DeFGs and QFGs, respectively. We proposed/re-proposed and analyzed generalized versions of belief-propagation algorithms for DeFGs/QFGs. As a particular application of the DeFGs, we investigate the information rate and their upper/lower bounds of classical communications over quantum channels with memory. In this study, we also propose a data-driven method for optimizing the upper/lower bounds on information rate.Comment: This is the finial version of the thesis of Michael X. Cao submitted in April 2021 in partial fulfillment of the requirements for the degree of doctor of philosophy in information engineering at the Chinese university of Hong Kon

New Algorithmic Paradigms for Discrete Problems using Dynamical Systems and Polynomials

Optimization is a fundamental tool in modern science. Numerous important tasks in biology, economy, physics and computer science can be cast as optimization problems. Consider the example of machine learning: recent advances have shown that even the most sophisticated tasks involving decision making, can be reduced to solving certain optimization problems. These advances however, bring several new challenges to the field of algorithm design. The first of them is related to the ever-growing size of instances, these optimization problems need to be solved for. In practice, this forces the algorithms for these problems to run in time linear or nearly linear in their input size. The second challenge is related to the emergence of new, harder and harder problems which need to be dealt with. These problems are in most cases considered computationally intractable because of complexity barriers such as NP completeness, or because of non-convexity. Therefore, efficiently computable relaxations for these problems are typically desired. The material of this thesis is divided into two parts. In the first part we attempt to address the first challenge. The recent tremendous progress in developing fast algorithm for such fundamental problems as maximum flow or linear programming, demonstrate the power of continuous techniques and tools such as electrical flows, fast Laplacian solvers and interior point methods. In this thesis we study new algorithms of this type based on continuous dynamical systems inspired by the study of a slime mold Physarum polycephalum. We perform a rigorous mathematical analysis of these dynamical systems and extract from them new, fast algorithms for problems such as minimum cost flow, linear programming and basis pursuit. In the second part of the thesis we develop new tools to approach the second challenge. Towards this, we study a very general form of discrete optimization problems and its extension to sampling and counting, capturing a host of important problems such as counting matchings in graphs, computing permanents of matrices or sampling from constrained determinantal point processes. We present a very general framework, based on polynomials, for dealing with these problems computationally. It is based, roughly, on encoding the problem structure in a multivariate polynomial and then recovering the solution by means of certain continuous relaxations. This leads to several questions on how to reason about such relaxations and how to compute them. We resolve them by relating certain analytic properties of the arising polynomials, such as the location of their roots or convexity, to the combinatorial structure of the underlying problem. We believe that the ideas and mathematical techniques developed in this thesis are only a beginning and they will inspire more work on the use of dynamical systems and polynomials in the design of fast algorithms

Sequential importance sampling for estimating expectations over the space of perfect matchings

This paper makes three contributions to estimating the number of perfect matching in bipartite graphs. First, we prove that the popular sequential importance sampling algorithm works in polynomial time for dense bipartite graphs. More carefully, our algorithm gives a $(1-\epsilon)$-approximation for the number of perfect matchings of a $\lambda$-dense bipartite graph, using $O(n^{\frac{1-2\lambda}{8\lambda}+\epsilon^{-2}})$ samples. With size $n$ on each side and for $\frac{1}{2}>\lambda>0$, a $\lambda$-dense bipartite graph has all degrees greater than $(\lambda+\frac{1}{2})n$. Second, practical applications of the algorithm requires many calls to matching algorithms. A novel preprocessing step is provided which makes significant improvements. Third, three applications are provided. The first is for counting Latin squares, the second is a practical way of computing the greedy algorithm for a card guessing game with feedback, and the third is for stochastic block models. In all three examples, sequential importance sampling allows treating practical problems of reasonably large sizes