Modern Coding Theory: The Statistical Mechanics and Computer Science Point of View

These are the notes for a set of lectures delivered by the two authors at the Les Houches Summer School on `Complex Systems' in July 2006. They provide an introduction to the basic concepts in modern (probabilistic) coding theory, highlighting connections with statistical mechanics. We also stress common concepts with other disciplines dealing with similar problems that can be generically referred to as `large graphical models'. While most of the lectures are devoted to the classical channel coding problem over simple memoryless channels, we present a discussion of more complex channel models. We conclude with an overview of the main open challenges in the field.Comment: Lectures at Les Houches Summer School on `Complex Systems', July 2006, 44 pages, 25 ps figure

A Simple Proof of Maxwell Saturation for Coupled Scalar Recursions

Low-density parity-check (LDPC) convolutional codes (or spatially-coupled codes) were recently shown to approach capacity on the binary erasure channel (BEC) and binary-input memoryless symmetric channels. The mechanism behind this spectacular performance is now called threshold saturation via spatial coupling. This new phenomenon is characterized by the belief-propagation threshold of the spatially-coupled ensemble increasing to an intrinsic noise threshold defined by the uncoupled system. In this paper, we present a simple proof of threshold saturation that applies to a wide class of coupled scalar recursions. Our approach is based on constructing potential functions for both the coupled and uncoupled recursions. Our results actually show that the fixed point of the coupled recursion is essentially determined by the minimum of the uncoupled potential function and we refer to this phenomenon as Maxwell saturation. A variety of examples are considered including the density-evolution equations for: irregular LDPC codes on the BEC, irregular low-density generator matrix codes on the BEC, a class of generalized LDPC codes with BCH component codes, the joint iterative decoding of LDPC codes on intersymbol-interference channels with erasure noise, and the compressed sensing of random vectors with i.i.d. components.Comment: This article is an extended journal version of arXiv:1204.5703 and has now been accepted to the IEEE Transactions on Information Theory. This version adds additional explanation for some details and also corrects a number of small typo

Spherical and Hyperbolic Toric Topology-Based Codes On Graph Embedding for Ising MRF Models: Classical and Quantum Topology Machine Learning

The paper introduces the application of information geometry to describe the ground states of Ising models by utilizing parity-check matrices of cyclic and quasi-cyclic codes on toric and spherical topologies. The approach establishes a connection between machine learning and error-correcting coding. This proposed approach has implications for the development of new embedding methods based on trapping sets. Statistical physics and number geometry applied for optimize error-correcting codes, leading to these embedding and sparse factorization methods. The paper establishes a direct connection between DNN architecture and error-correcting coding by demonstrating how state-of-the-art architectures (ChordMixer, Mega, Mega-chunk, CDIL, ...) from the long-range arena can be equivalent to of block and convolutional LDPC codes (Cage-graph, Repeat Accumulate). QC codes correspond to certain types of chemical elements, with the carbon element being represented by the mixed automorphism Shu-Lin-Fossorier QC-LDPC code. The connections between Belief Propagation and the Permanent, Bethe-Permanent, Nishimori Temperature, and Bethe-Hessian Matrix are elaborated upon in detail. The Quantum Approximate Optimization Algorithm (QAOA) used in the Sherrington-Kirkpatrick Ising model can be seen as analogous to the back-propagation loss function landscape in training DNNs. This similarity creates a comparable problem with TS pseudo-codeword, resembling the belief propagation method. Additionally, the layer depth in QAOA correlates to the number of decoding belief propagation iterations in the Wiberg decoding tree. Overall, this work has the potential to advance multiple fields, from Information Theory, DNN architecture design (sparse and structured prior graph topology), efficient hardware design for Quantum and Classical DPU/TPU (graph, quantize and shift register architect.) to Materials Science and beyond.Comment: 71 pages, 42 Figures, 1 Table, 1 Appendix. arXiv admin note: text overlap with arXiv:2109.08184 by other author

Iterative graphical algorithms for phase noise channels.

Doctoral Degree. University of KwaZulu-Natal, Durban.This thesis proposes algorithms based on graphical models to detect signals and charac- terise the performance of communication systems in the presence of Wiener phase noise. The algorithms exploit properties of phase noise and consequently use graphical models to develop low complexity approaches of signal detection. The contributions are presented in the form of papers. The first paper investigates the effect of message scheduling on the performance of graphical algorithms. A serial message scheduling is proposed for Orthogonal Frequency Division Multiplexing (OFDM) systems in the presence of carrier frequency offset and phase noise. The algorithm is shown to have better convergence compared to non-serial scheduling algorithms. The second paper introduces a concept referred to as circular random variables which is based on exploiting the properties of phase noise. An iterative algorithm is proposed to detect Low Density Parity Check (LDPC) codes in the presence of Wiener phase noise. The proposed algorithm is shown to have similar performance as existing algorithms with very low complexity. The third paper extends the concept of circular variables to detect coherent optical OFDM signals in the presence of residual carrier frequency offset and Wiener phase noise. The proposed iterative algorithm shows a significant improvement in complexity compared to existing algorithms. The fourth paper proposes two methods based on minimising the free energy function of graphical models. The first method combines the Belief Propagation (BP) and the Uniformly Re-weighted BP (URWBP) algorithms. The second method combines the Mean Field (MF) and the URWBP algorithms. The proposed methods are used to detect LDPC codes in Wiener phase noise channels. The proposed methods show good balance between complexity and performance compared to existing methods. The last paper proposes parameter based computation of the information bounds of the Wiener phase noise channel. The proposed methods compute the information lower and upper bounds using parameters of the Gaussian probability density function. The results show that these methods achieve similar performance as existing methods with low complexity