24 research outputs found
The Generalized Area Theorem and Some of its Consequences
There is a fundamental relationship between belief propagation and maximum a
posteriori decoding. The case of transmission over the binary erasure channel
was investigated in detail in a companion paper. This paper investigates the
extension to general memoryless channels (paying special attention to the
binary case). An area theorem for transmission over general memoryless channels
is introduced and some of its many consequences are discussed. We show that
this area theorem gives rise to an upper-bound on the maximum a posteriori
threshold for sparse graph codes. In situations where this bound is tight, the
extrinsic soft bit estimates delivered by the belief propagation decoder
coincide with the correct a posteriori probabilities above the maximum a
posteriori threshold. More generally, it is conjectured that the fundamental
relationship between the maximum a posteriori and the belief propagation
decoder which was observed for transmission over the binary erasure channel
carries over to the general case. We finally demonstrate that in order for the
design rate of an ensemble to approach the capacity under belief propagation
decoding the component codes have to be perfectly matched, a statement which is
well known for the special case of transmission over the binary erasure
channel.Comment: 27 pages, 46 ps figure
Decay of Correlations for Sparse Graph Error Correcting Codes
The subject of this paper is transmission over a general class of
binary-input memoryless symmetric channels using error correcting codes based
on sparse graphs, namely low-density generator-matrix and low-density
parity-check codes. The optimal (or ideal) decoder based on the posterior
measure over the code bits, and its relationship to the sub-optimal belief
propagation decoder, are investigated. We consider the correlation (or
covariance) between two codebits, averaged over the noise realizations, as a
function of the graph distance, for the optimal decoder. Our main result is
that this correlation decays exponentially fast for fixed general low-density
generator-matrix codes and high enough noise parameter, and also for fixed
general low-density parity-check codes and low enough noise parameter. This has
many consequences. Appropriate performance curves - called GEXIT functions - of
the belief propagation and optimal decoders match in high/low noise regimes.
This means that in high/low noise regimes the performance curves of the optimal
decoder can be computed by density evolution. Another interpretation is that
the replica predictions of spin-glass theory are exact. Our methods are rather
general and use cluster expansions first developed in the context of
mathematical statistical mechanics.Comment: 40 pages, Submitted to SIAM Journal of Discrete Mathematic
Conservation laws for coding
This work deals with coding systems based on sparse graph codes. The key issue we address is the relationship between iterative (in particular belief propagation) and maximum a posteriori decoding. We show that between the two there is a fundamental connection, which is reminiscent of the Maxwell construction in thermodynamics. The main objects we consider are EXIT-like functions. EXIT functions were originally introduced as handy tools for the design of iterative coding systems. It gradually became clear that EXIT functions possess several fundamental properties. Many of these properties, however, apply only to the erasure case. This motivates us to introduce GEXIT functions that coincide with EXIT functions over the erasure channel. In many aspects, GEXIT functions over general memoryless output-symmetric channels play the same role as EXIT functions do over the erasure channel. In particular, GEXIT functions are characterized by the general area theorem. As a first consequence, we demonstrate that in order for the rate of an ensemble of codes to approach the capacity under belief propagation decoding, the GEXIT functions of the component codes have to be matched perfectly. This statement was previously known as the matching condition for the erasure case. We then use these GEXIT functions to show that in the limit of large blocklengths a fundamental connection appears between belief propagation and maximum a posteriori decoding. A decoding algorithm, which we call Maxwell decoder, provides an operational interpretation of this relationship for the erasure case. Both the algorithm and the analysis of the decoder are the translation of the Maxwell construction from statistical mechanics to the context of probabilistic decoding. We take the first steps to extend this construction to general memoryless output-symmetric channels. More exactly, a general upper bound on the maximum a posteriori threshold for sparse graph codes is given. It is conjectured that the fundamental connection between belief propagation and maximum a posteriori decoding carries over to the general case
Statistical physics methods for sparse graph codes
This thesis deals with the asymptotic analysis of coding systems based on sparse graph codes. The goal of this work is to analyze the decoder performance when transmitting over a general binary-input memoryless symmetric-output (BMS) channel. We consider the two most fundamental decoders, the optimal maximum a posteriori (MAP) decoder and the sub-optimal belief propagation (BP) decoder. The BP decoder has low-complexity and its performance analysis is, hence, of great interest. The MAP decoder, on the other hand, is computationally expensive. However, the MAP decoder analysis provides fundamental limits on the code performance. As a result, the MAP-decoding analysis is important in designing codes which achieve the ultimate Shannon limit. It would be fair to say that, over the binary erasure channel (BEC), the performance of the MAP and BP decoder has been thoroughly understood. However, much less is known in the case of transmission over general BMS channels. The combinatorial methods used for analyzing the case of BEC do not extend easily to the general case. The main goal of this thesis is to advance the analysis in the case of transmission over general BMS channels. To do this, we use the recent convergence of statistical physics and coding theory. Sparse graph codes can be mapped into appropriate statistical physics spin-glass models. This allows us to use sophisticated methods from rigorous statistical mechanics like the correlation inequalities, interpolation method and cluster expansions for the purpose of our analysis. One of the main results of this thesis is that in some regimes of noise, the BP decoder is optimal for a typical code in an ensemble of codes. This result is a pleasing extension of the same result for the case of BEC. An important consequence of our results is that the heuristic predictions of the replica and cavity methods of spin-glass theory are correct in the realm of sparse graph codes
Sharp Bounds for Optimal Decoding of Low Density Parity Check Codes
Consider communication over a binary-input memoryless output-symmetric
channel with low density parity check (LDPC) codes and maximum a posteriori
(MAP) decoding. The replica method of spin glass theory allows to conjecture an
analytic formula for the average input-output conditional entropy per bit in
the infinite block length limit. Montanari proved a lower bound for this
entropy, in the case of LDPC ensembles with convex check degree polynomial,
which matches the replica formula. Here we extend this lower bound to any
irregular LDPC ensemble. The new feature of our work is an analysis of the
second derivative of the conditional input-output entropy with respect to
noise. A close relation arises between this second derivative and correlation
or mutual information of codebits. This allows us to extend the realm of the
interpolation method, in particular we show how channel symmetry allows to
control the fluctuations of the overlap parameters.Comment: 40 Pages, Submitted to IEEE Transactions on Information Theor