The "turbo codes", recently proposed by Berrou et. al. are written as a
disordered spin Hamiltonian. It is shown that there is a threshold Theta such
that for signal to noise ratios v^2 / w^2 > Theta, the error probability per
bit vanishes in the thermodynamic limit, i.e. the limit of infinitly long
sequences. The value of the threshold has been computed for two particular
turbo codes. It is found that it depends on the code. These results are
compared with numerical simulations.Comment: 23 pages, 6 figures: Fig.2 has been replaced (in the preceding
  version it was identical to Fig.1

Montanari, A.

Sourlas, N.

English

arXiv

この論文は国立情報学研究所の電子図書館事業により電子化されました。最近,Berrou等によって提案された"ターボ符号"は不規則系のスピンハミルトニアンによって記述することが出来る.この符号では熱力学極限,すなわち長符号長の極限で,SN比がv^2/w^2がある閾値Θ以上ならば1ビット当たりの誤り確率がゼロとなることが示される.我々は2つの典型的なターボ符号に関してこの閾値を計算した.その結果,閾値は個々の符号の構成法に依存することが明らかになった.さらにこれを数値実験と比較した.なお,これらはAndrea Montanariとの共同研究によって得られた結果である.The "turbo codes", recently proposed by Berrou et. al. are written as a disordered spin Hamiltonian. It is shown that there is a threshold Θ such that for signal to noise ratios v^2/w^2 > Θ the error probability per bit vanishes in the thermodynamic limit, i.e. the limit of infinitly long sequences. The value of the threshold has been computed for two particular turbo codes. It is found that it depends on the code. These results are compared with numerical simulations. This work was done in collaboration with Andrea Montanari

Sourlas, Nicolas

Kyoto University Research Information Repository

Title The statistical mechanics of turbo codesAuthor(s)Sourlas, NicolasCitation物性研究 (2000), 73(5): 811-819Issue Date2000-02-20URL http://hdl.handle.net/2433/96788RightType Departmental Bulletin PaperTextversionpublisherKyoto UniversityThe statistical mechanics of turbo codesINicolas Sourlas~i!i:, Berrou ~~: J: -::> -C1~~ ~ nt:: "7 - *-r~-ry-" '±/F~J~JW*O) A e:~ J\ ~ )v r =-7 ~~: J: -::> -C~G~'t ~ ~ t jQ'i'ttl*~. ~ O)r~-ry-'t"'±~:1J~*,j~l, 't~ b -t ffi:r~-ry-ffi:O)*,j~I1.", SN Jt V2/W2 jQ'i'~ ~~1WI e £tl:.~ G';f' 1 ~- /' r ~ t:: 0 0)~ 0it$ jQ'i'~ tJ t ~ ~ ~ t jQ'i'~~n ~. ~ 4 ,± 2 "JO)~~B9~7 -*-r~-ry-~:~L -C ~ 0)~1WI~at.Lt:: . .z-O)*a*, ~1WI'±11m 4 O)r~-ry-O)fltnX:t:t~:1tt1¥'t ~ ~ t jQ'i'OO GjQ~~: ~ -::> t::. ~ G~: ~n ~ it:~~,~~ t ltfX L t::. ~ J3, ~ n G,± Andrea Montanari t 0)~/PJ1iJfJE~:J: -::> -C 1~Gn t:: *a*'t"~ ~ .- 811-The statistical mechanics of turbo codes.Nicolas SourlasLaboratoire de Physique Theorique de l' Ecole Normale Superieure *24 rue Lhomond, 75231 Paris CEDEX OS, France.email: sourlas@lpt.ens.frAbstractThe "turbo codes", recently proposed by Berrou et. al. [1] are written as adisordered spin Hamiltonian. It is shown that there is a threshold e such thatfor signal to noise ratios v2 / w2 > e the error probability per bit vanishes in thethermodynamic limit, i.e. the limit of infinitly long sequences. The value of thethreshold has been computed for two particular turbo codes. It is found that itdepends on the code. These results are compared with numerical simulations. Thiswork was done in collaboration with Andrea Montanari [8].The recent invention of "turbo codes" by Berrou and Glavieux [1] is considered amajor breakthrough in communications. For the first time one can communicate almosterror-free for signal to noise ratios very close to the theoretical bounds of informationtheory. Turbo codes are fastly becoming the new standard for error correcting codes indigital communications. The invention of turbo codes and their iterative decoding algo-rithm was empirical. There is no theoretical understanding of why they are so successfull.The decoding algorithm is thought to be an appro~imatealgorithm. We think that turbocodes are interesting, even outside the context of communication theory, because theyprovide a non trivial example of a disordered system which can be studied numericallywith a fast algorithm.In this paper we will study turbo codes and turbo decoding using the modern tools ofstatistical mechanics of disordered systems. We have already shown in the past [4] thatthere is a mathematical equivalence between error correcting codes and theoretical modelsof spin glasses. In particular the logarithm of the probability for any given signal, condi-tional on the communication channel output, has the form of a spin glass Hamiltonian.*UMR 8549, Unite Mixte de Recherche du Centre National de la Recherche Scientifique et de l' EcoleNormale Superieure.- 812-We will construct the Hamiltonian which corresponds to the turbo codes and study itsproperties. This will clarify why they are so successfull. In particular we will show thatthere is a threshold e such that for signal to noise ratios v2 / w 2 > e the average errorprobability per bit Pe vanishes in the thermodynamic limit, i.e. the limit of infinitly longsequences. In Pe the average is taken over a large class of turbo codes (see later) and over"channel" noise. The value of the threshold has been computed for two particular turbocodes. It was found that it depends on the code. We also compare these results withnumerical simulations.We consider this result as particularly interesting. They were only two known familiesof codes which can achieve zero error asymptotically: a) orthogonal codes and b) codesbased on Derrida's random energy model. Both families become error-free only in thezero rate limit i.e. the limit of infinite redundancy (see later). Turbo codes are the firstfinite rate codes to be shown to permit error-free communication.We consider this paper as a new application of statistical mechanics to a problem outsideits original domain of applicability. Our results are typical of the statistical mechanicsapproach: we study only the average performance of turbo codes, not the performanceof any particular one. Furthermore there exist (very "few") particular codes (the onescorresponding to permutations close to the identity, see later), performing much worsethan the average. For those codes the error probability per bit Pe # 0, but they are so"few" that this does not prevent the average Pe from vanishing.Let us first briefly remind the connection between error-correction codes and spin-glassmodels. In the mathematical theory of communication both the production of informa-tion and its transmission are considered as probabilistic events. A source is producinginformation messages according to a certain probability distribution. Messages of lengthN are sequencies of N symbols or "letters of an alphabet" aI, a2, ... , aN. We will assumefor simplicity a binary alphabet, i.e. ai = 0 or 1 and that all symbols are equally probable.Instead of ai we can equally well use Ising spins(1)The messages are sent through a noisy transmission channel. If a (J = ± 1 is sent throughthe transmission channel, because of the noise, the output will be a real number (J0ut, ingeneral different from (J. Again, the statistical properties of the transmission channel aresupposed to be known. Let us call Q(u out 10' )duout the probability for the transmissionchannel's output to be between u out and uout+duout, when the input was 0'. Q(u out 10')is supposed to be known. We assume that the noise is independent for any pair of bits("memoryless channel"), i.e.(2)In the case of a memoryless channel and a gaussian noise:(3)- 813-Shannon calculated the channels capacity C, i.e. the maximum information per use of thechannel that can be transmitted.(4)where v2 is the signal power.Under the above assumptions, communication is a statistical inference problem. Giventhe transmission channel's output and the statistical properties of the source and of thechannel, one has to infer what message was sent. In order to reduce communication errors,one may introduce (deterministic) redundancy into the message ("channel encoding") anduse this redundancy to infer the message sent through the channel ("decoding"). Thealgorithms which transform the source outputs to redundant messages are called error-correcting codes. More precisely, instead of sending the N original bits ai, one sends Mbits Jtn , k = 1,"', M,M > N, constructed in the following way(5)where the "connectivity" matrix Ci(~~.ilk has elements zero or one. For any k, all the Ci(~~.ilkexcept from one are equal to zero, i.e. the Jtn are equal to ±l. Ci(~.~.ilk defines the code,i.e. it tells from which of the a's to construct the kth bit of the code.This kind of codes are called parity checking codes because J~n counts the parity of theminusis among the lk a's. The ratio R = N / M which specifies the redudancy of the code,is called the rate of the code.Knowing the source probability, the noise probability, the code and the channel output,. one has to infer the message that was sent. The quality of inference depends on the choiceof the code.According to the famous Shannon's channel encoding theorem, there exist codes which,in the limit of infinitly long messages, allow error-free communication, provided the rateof the code R is less than the channel capacity C. This theorem says that such "ideal"codes exist, but does not say how to construct them.We have shown that there exists a close mathematical relationship between error-correctingcodes and theoretical models of disosdered systems. As we previously said, the output ofthe channel is a sequence of M real numbers Jout = {Jfut, k = 1,' . " M}, which arerandom variables, obeying the probability distribution Q(JfutIJ~n). Once the channeloutput Jout is known, it is possible to compute the probability P(rlJout ) for any par-ticular sequence r = {Ti' i = 1,"', N} to be the source output (i.e. the informationmessage).More precisely, the equivalence between spin-glass models and error correcting codes isbased on the following property.The probability P(rlJout ) for any sequence r to be the information message, conditionalon the channel output Jout is given byMIn P(rJJout) = const +L Cf~~.ilk B k Til' .. Tilkk=l- 814-~H(r) (6)where(7)We recognize in this expression the Hamiltonian of a p-spin spin-glass Hamiltonian. Thedistribution of the couplings is determined by the probability Q(JoutIJin).In the case when Q(JoutIJin) = Q(_Joutl_ Jin) (the case of a "symmetric channel"),B(Jout ) = -Bk ( _Jout ) and one recovers the invariance of the spin-glass Hamiltonianunder gauge transformations."Minimum error probability decoding" (or MED), which is widely used in communications[2], consists in choosing the most prob9-ble sequence rO. This is equivalent to finding theground state of the above spin-glass Hamiltonian.Instead of considering the most probable instance, one may only be interested in the mostprobable value TrAP of the "bit" Ti (Maximum A posteriori Probability or MAP decoding)[3] which can be expressed in terms of the magnetization at temperature T= 1/f3 equalto one [6]:~ I:Ti exp{ -H(r)}r(8)where H(r) is defined by Eq.(6).It is remarkable that f3 = 1 coincides with the Nishimori temperature in spin glasses [7].MAP decoding is an essential ingredient in turbo decoding (see later).When all messages are equally probable and the transmission channel is memoryless andsymmetric, the error probability is the same for all input sequences. It is enough tocompute it in the case where all input bits are equal to one. In this case, the errorprobability per bit Pe is1 (d) _ 1 ( 1 ~ (d))Pe = 2(1 - m ) = 2 1 - N ~ Tiz=l(9)and Ti(d) is the symbol sequence produced by the decoding procedure. One can derive fromthis a very general lower bound for Pe , using the analog of the low temperature expansion.An obvious bound (for zero temperature decoding) is provided by the probability pP) thatonly one bit is incorrect, i.e. Tj = -1 while all other bits are correct, i.e. Ti = 1 for alli i- j:Pe ;::: p~1) = Probability of { I: B k < o}kEr2(j)(10)where the O(j) denotes the set of the couplings in which Tj appears.A necessary condition for transmitting without errors is that LkEr2U) B k > 0 with prob-ability one. This is only possible if every spin appears in an infinite number of terms in- 815-the Hamiltonian. Let lk be the number of spins coupled through the coupling Bk .The total number of spins beeing N, a spin appears on the average in(11 )terms, where [ is the average of lk (the number of spins coupled together) and R is therate of the code.So a necessary condition for a finite rate code to achieve zero error probability, is that theaverage number of spins coupled together diverges in the thermodynamic limit (N --+ 00).This condition is realised in Derrida's random energy model [5] which has been shown tobe an ideal code [4] ( in that case R = °).We will show in the following that this is also true for the case of recursive turbo codes,while it is not true for non recursive turbo codes.We first present a review of convolutional codes. Convolutional codes are the buildingblocks of turbo codes. We shall describe both non recursive and recursive convolutionalcodes. They correspond to one-dimensional spin models. The information message willbe denoted by:(12)It is convenient to think of the source producing a symbol per unit time, i.e. in Ti, idenotes the time. For simplicity we consider a code of rate R = 1/2. The encodedmessage has the form:_ ( (1) (1). (2) (2))J = J1 , ..• , JN ,J1 , ... , JN (13)Any hardware implementation of a convolutional encoder contains a sequence of r memoryregisters. We shall call r the range of the code.Let's denote by ~l(t), ... , ~r(t) the content of the memory registers at time t. At eachtime step the content of the memory register is shifted to the right:~j+1(t+ 1) = ~j(t) for j = 1, ... , r - 1, ~o(t) - ~l(t + 1)We define the following sequence of bits which we shall call the register sequence:IT=(Ol, ... ,ON), oi-~o(i)For non recursive convolutional codes,(14)(15)The encoded message J is easily defined in terms of the content of the register sequence:r rJi(n) = IT(~j(i)t(j;n) = IT(Oi_jt(j;n)j=O j=Oi = 1, ... , N ; n = 1,_2K,(j; n) E {O, I}- 816-(16)The numbers K-(j; n) define the code.Recursive convolutional codes are most easily defined byr rai(T) = Eo(i) = Ti II Ej (i)K(jjl) = Ti II(ai_j)K(jjl)j=l j=l(17)We shall now consider decoding. The probability distribution of the register sequenceconditional to some ouput can be written as the Boltzmann weight of a spin model withrandom couplings. The Hamiltonian is given by:(18)where B(·) is defined in Eq.(7).We define the decoding at arbitrary temperature T - 1/(3 as follows:sign((Ti (U ) ) fJ )1 t "" O(u) exp{ -(3H(u; JoutnZ(Jou ;(3) 7(19)(20)where the expression for Ti(U) is given by Eq.(17) or by Eq.(15) depending whether thecode is recursive or not.As seen in the introduction there are two widely used decoding strategies:• Maximum Likelihood decoding which consists in finding the most probable sequenceof bits and corresponds to the choice f3 = 00 in Eq.(19): TiML Tf=oo .• Maximum A posteriori Probability decoding which consists in finding the mostprobable sequence of bits and corresponds to the choice (3 = 1 in Eq.(19): TiMAP =Tf=l. This is the strategy which enters in turbo decoding.Both these strategies can be implemented in a very efficient way using the transfer matrixtechnique. The corresponding algorithms are known in communication theory as theViterbi algorithm [2] for the (3 = 00 case and the BCJR algorithm [3] for the (3 = 1 case.The complexity of these algorithms grows like N2r .A turbo code is defined by the choice of a convolutional code and of a permutation of Nobjects. We use for the permutation the following notation:P:{I, ... ,N} --t {1, ... ,N}, if----tP(i) (21)The basic idea is to apply the permution P to the source sequence T to produce a newsequence T P . Both sequences T and T P are the inputs to two set of registers, each oneimplementing a convolutional encoding. In this way the rate of the code is decreased (i.e.greater redundancy).- 817-We illustrate this idea with the example of a rate 1/2 recursive convolutional code, definedby the constants K(j; 1) and K(j; 2). The two register sequences are:rail) - ai(T) => Ti = II(ai-=:jt(jj1) - Ei(U(1))j=lra?) - ai(TP) => Tt = II(ai~jt(jj1) _ Ei(U(2))j=l(22)(23)where T P is the permuted message (Tt - TP(i))'The relation between the two register sequences is rather involved and nonlocal for a gen-eral choice of the permutation. Moreover ap) can be expressed only in terms of a largenumber of ay) 'SoIt turns out that it is convenient to write the corresponding Hamiltonian as a functionof both register sequences. This introduces new degrees of freedom and the Hamiltonianis a function of 2N instead of N spin. The unwanted degrees of freedom are eliminatedby imposing the constraint Tt = Tp(i). This constraint can be written in terms of the a'susing Eqs.(22) and (23). In this way the probability distribution is a local function of thespin variables a(1) and a(2).Solving the constraint produces an infinite connectivity Hamiltonian for recursive turbocodes. For non recursive codes, the connectivity remains finite. This finite versus infiniteconnectivity is the essential difference between non recursive and recursive turbo codesand explains why recursive turbo codes are so better and why they can achieve zero errorprobability for low enough noise.We now discuss decoding. There is no exact decoding algorithm for turbo codes. Berrou etal. have proposed a very ingenious algorithm, called turbo decoding, which is thought tobe approximate. Turbo decoding is an iterative procedure. At each step of the iteration,one considers one of the two chains and proceeds to MAP decoding. The information soobtained is injected to the next step by adding appropriate external fields to the Hamil-tonian of the other chain. The algorithm terminates if a fixed point is reached.Turbo decoding can be seeing as the union of two one dimensional subsystem. Eachsubsystem acts on the other one through a magnetic field (in the non recursive case) orthrough an additional coupling (in the recursive case).We would like to compute the error probability per bit. As explained above, in the caseof a symmetric transmission channel, it is enough to compute the magnetization in thecase of all inputs Ti = 1.The similarity of the Hamiltonian of turbo codes with the Hamiltonians of disordered spinsystems is obvious. The disorder in the case of turbo codes has two origins. One is due tothe (random) permutation which defines the particular code. The other is more conven-tional and is related to the randomness of the couplings which is due to· the transmissionnoise. As usual in disordered systems, we can only compute the average over disorder andfor that we have to introduce replicas. We briefly report the main results of this approach[8] for the gaussian channel described by Eq. (3).For recursive turbo codes there exists a low noise phase w2 < w~ where the error prob-ability vanishes in the thermodynamic limit (Le. for infinitely long sequences). In this- 818-phase the model is completely ordered. A local stability analysis yields the critical valuewloc such that for w2 > wloc the no-error phase is destroyed by small fluctuations. Clearlywloc ~ w~. We computed wfoc for the two cases of turbo codes.It is well known that the replica method is not mathematically rigorous. So it is naturalto question the validity of our results. For this purpose we have carried out numericalsimulations. We used the Berrou et al. turbo decoding algorithm and averaged over 200to 500 realizations of the disorder.The first conclusion is that recursive turbo codes are much better codes that non recursiveones. Furthemore our results for recursive turbo codes are compatible with the existenceof a threshold w~ such that for w2 < w~ the error probability per bit is zero, while no suchthreshold seems to exist for non recurs~ve codes. This is in agreement with replica theory.Zero error probability can only be achieved in the N ~ 00 limit. OUf simulations are forN = 105 . It would be interesting to perform a detailed study of finite size corrections, i.e.of the N dependance of the error probability per bit.Another important issue is the breaking of replica symmetry. Since turbo-decoding isthought to be an approximate algorithm, it may be not the best tool to look for replicasymmetry breaking. We have started an analytical investigation of replica symmetrybreaking.References[1] C.Berrou, A.Glavieux, and P.Thitimajshima. Proc.1993 lnt.Conf.Comm. 1064-1070[2] A.J.Viterbi. IEEE Trans.Com.Technology COM-19(1971) 751-771[3] L.Bahl, J.Cocke, F.Jelinek, and J.Raviv. IEEE Trans.Inf.Theory IT-20(1974) 248-287[4] N.Sourlas. Nature 339(1989) 693-694N.Sourlas, in Statistical Mechanics of Neural Networks, Lecture Notes in Physics 368,ed. L. Garrido, Springer Verlag (1990)N.Sourlas, Ecole Normale Superieure preprint (April 1993)N.Sourlas, in From Statistical Physics to Statistical Inference and Back, ed. P. Grass-berger and J.-P. Nadal, Kluwer Academic (1994), page 195.[5] B.Derrida. Phys.Rev. B 24(1981) 2613-2626[6] P.Rujan. Phys.Rev.Lett. 70(1993) 2968-2971N.Sourlas. Europhys.Lett. 25(1994) 159-164H.Nishimori. J. Phys. Soc. Jpn. 62(1993) 2973[7] H.Nishimori. J.Phys. C 13(1980) 4071-4076[8] A.Montanari and N.Sourlas, to appear.A.Montanari, in preparation.- 819-

The statistical mechanics of turbo codes

The "turbo codes", recently proposed by Berrou
et al. [1] 
are written as a disordered spin Hamiltonian. It is shown 
that there exists a threshold Θ such that 
for signal to noise ratios $1 / w^2 >  \Theta$ the error probability per 
bit vanishes in the thermodynamic limit, i.e. the limit of infinitely 
long sequences. The value of the threshold has been computed for 
two particular turbo codes. It is found that it depends on the code. These 
results are compared with numerical simulations

The statistical mechanics of turbo codes

Abstract

Similar works

Full text

Available Versions

Kyoto University Research Information Repository

EDP Sciences OAI-PMH repository (1.2.0)

Crossref