Contains reports on four research projects.Lincoln Laboratory (Purchase Order B-00306)United States ArmyUnited States NavyUnited States Air Force (Contract AF19(604)-7400

Cunningham, J. E.

Gronemann, Uri F.

Huang, T. S.

Pan, J. W.

Schreiber, William F.

Tretiak, O. J.

Ziv, J.

DSpace@MIT

IX. PROCESSING AND TRANSMISSION OF INFORMATIOND. N. ArdenE. ArthursJ. B. DennisM. EdenP. EliasR. M. FanoR. G. GallagerE. M. HofstetterD. A. HuffmanW. F. SchreiberC. E. ShannonProf. J. M. WozencraftS. AsanoG. CheadleJ. E. CunninghamP. M. EbertH. A. ErnstE. F. FerrettiT. J. Goblick, Jr.U. F. GronemannF. C. Hennie IIIS. HuangJelinekKailathKleinrockH. Loomis, Jr.L. MasseyW. PanL. ReichG. RobertsJ. TretiakZivA. PICTURE PROCESSING RESEARCHWe have now made computer tapes from original pictures and played the tapes backfor photography. Operations on the IBM 709 computer will begin during the nextreporting period.An example of a 1024 X 1024 element picture is shown in Fig. IX-1.Fig. IX-1.J. E. Cunningham, U. F. Gronemann, T. S. Huang,J. W. Pan, O. J. Tretiak, W. F. Schreiber*This research was supported in part by Purchase Order DDL B-00306 with LincolnLaboratory, a center for research operated by Massachusetts Institute of Technology,with the joint support of the U.S. Army, Navy, and Air Force under Air Force ContractAF19(604)-7400.119Prof.Prof.Prof.Prof.Prof.Prof.Prof.Prof.Prof.Prof.Prof.(IX. PROCESSING AND TRANSMISSION OF INFORMATION)B. NEW SYSTEMATIC DECODING FOR MEMORYLESS CHANNELSIn this report a sequential decoding scheme for random convolutional codes, whichis to be used for memoryless channels, is described. The average number of compu-tations does not grow exponentially with n; it is upper-bounded by a quantity proportional2to n , for all rates below some cutoff rate R (n is the block length).When this decoding scheme is averaged over a suitably defined ensemble of codewords it has an average probability of error with an upper bound whose logarithm is-nE(R). E(R) is dependent on the data rate. (E(R) > 0 for rates below channel capacity.)The decoding scheme is different from other effective decoding schemes such as sequen-1 2tial decoding and low-density parity-check codes. 2The lower bound on R comp of the systematic decoding scheme that is presented inthis report is the same as the Rcomp of sequential decoding for asymmetric channels.However, in the case of sequential decoding, R is valid only for the incorrectcompsubset of code words. The existence of Rcomp for the correct subset has not yet beenestablished.Thus, the systematic decoding scheme yields a better bound on the average numberof computations for asymmetric channels. (This is not the case when the channel is sym-metric, since the modified sequential decoding scheme after Gallager 3 may be used.)S* * Fig. IX-2. A convolutional tree code.A convolutional tree code may be viewed topologically as shown in Fig. IX-2. Thewords are all the directed paths from the input node to the output nodes of the tree (thereare no closed paths). From all nontrivial intermediate nodes there emerge f directedlinks, one for each of I nodes. Let the number of input symbols per code word be n.Let the number of input symbols per link be d. Then the number of links per word ism = n/d (m is the number of information digits per word). In Fig. IX-2, n = 6; m = 2;d= 3; f = 2.Reiffen 4 has shown that the convolutional codes may be generated sequentially.1. The Decoding ProcedureThe decoding procedure consists of the following successive operations:Step 1: The a posteriori probability of each one of the I links of length d that120(IX. PROCESSING AND TRANSMISSION OF INFORMATION)emerge from the input node to the first k nodes in the tree is computed. The one linkthat yields the largest a posteriori probability is chosen to represent the correspondingpart of the transmitted code word. This detected link connects the input node with oneof the f nodes of the next set of nodes (set I in Fig. IX-3).The same procedure is then repeated with the detected node of set I as a startingpoint. Thus, the a posteriori probability of each one of the f links emerging from thed * Fig. IX-3. The decoding procedure of step 1.dI IIN I I Tnode that was previously detected, is now computed, and a decision is made. This pro-cedure is then repeated again and again until termination (i. e., until the detected pathreaches one of the output nodes). A metric of the form D(u, v) = In P(v/u) is then com-L P(v)puted. Here, P(v/u) is the a posteriori probability of the complete detected word, u,and P(v) is the probability of the output symbol v.If D(u, v) is larger than some preset threshold Do , a final decision is made and thedetection of the first information digit is completed. If D(u, v) < D o , the computationprocedure is then to go to step 2.2Step 2: The a posteriori probability of each one of the 1 links of length 2d thatemerge from the input node to set II (that consist of f2 nodes) is computed (Fig. IX-4).The one link that yields the largest a posteriori probability is chosen to represent thecorresponding part of the transmitted code word.The same procedure is then repeated with the detected node of set I as a startingpoint, and so on. This procedure is continued until termination (i. e., until the detectedpath reaches one of the output nodes).121(IX. PROCESSING AND TRANSMISSION OF INFORMATION)D(u, v) is then computed for the detected path and compared with the threshold D .oIf D(u, v) > Do, a final decision is made with respect to the first information digit. IfFig. IX-4. The decoding procedure of step 2.I II I II I II II I II I II II I II I II II It XID(u, v) < D O the computation procedure is then to go to step 3.Step 3: This step is similar to step 2, except that the computation length in thiscase is 3d, and the number of links involved at each stage is . If no terminationoccurs at step 3, the computation procedure then reaches step 4, and so on. Followingthe detection of the first information digit, the whole procedure is repeated for the nextdigit, and so forth.a. The Average Number of Computations per Information DigitThe number of computations that are involved in step 1 is equal to m . . Thenumber of computations in step 2 is equal to (m. 2). The number of computations instep 3 is equal to (m.3 ).In general, the number of computations in step k isNk = mfkLet Ck be the condition that no termination occur at step k. Step k will be usedonly if there were no terminations at all in the previous k - 1 steps. Thus the proba-bility of step k being used isP(k) = Pr(C1 ,C 2' C2, 3 C4. Ck-)122(IX. PROCESSING AND TRANSMISSION OF INFORMATION)The average number of computations is given byN = N 1P(1 ) + N 2 P(2) + ... + NkP(k) + ... NmP(m)m m= NkP(k) = I mfkP(k). (2)k=l k= 1b. An Upper Bound on the Average Number of ComputationsP(k) may be bounded byP(k) = Pr(C1, C2 C3 C4 ' Ck-l)< Pr(Ck- 1). (3)ThusmN= N Pr(Ck-1) = (m-k+l) Ik r(Ck-l1) (4)k= 1 k= 1Now let uk be the code word detected at step k, and let u be the transmitted code word.ThenPr(Ckl) = Pr(D(uk, v).< Do)= Pr(D(uk' v) Do;uk=u) + Pr(D(uk, v) 2< Do;Uku)= Pr(D(u, v) < Do;uk=u) +Pr(D(uk, v) -< Do;Uk#u)ThusPr(Ckl) - Pr[D(u,v) <Do] + Pr[uku] (5)The threshold D is set so as to make0Pr[D(u, v)-<DO] < e - (R)n, (6)where -E(R) is a function of the rate R.The number of detected links needed to construct uk is, as we have shown, (m-k+1).Now, u is equal to uk only if all of the (m-k+1) links of uk are equal to the correspondinglinks of u.k-1Let e. be the condition that the a posteriori probability of one of the (f-l)k links1 th themerging from the i node of u and not including the (i+l) node of u is greater orequal to that of the corresponding link of u. Then123(IX. PROCESSING AND TRANSMISSION OF INFORMATION)Pr(Uk=U) = Pr(not el; not e 2 ; not e 3; ... ; not e i ... ; not em-k+1)Pr(uk*u) = 1 - P r(uk=u) = Pr UL i=1The probability of the union of events is upper-bounded by the sum of the probabil-ities of the individual events. ThusPr(uk*u) -<m-k+ 1z Pr(ei)The rate per symbol is defined asR In M 1 In m In /d In 1n n n dwhere M is the number of code words of length n. Thus1R = - In IdFano has shown that for a given rate R and a given word length n the probabilityof error is bounded byP(e) < 2 e opt(R). n(9)(8)E(R) , is the optimum exponentthe rate R.of the average probability of error and is a function ofNow, in the case of Pr(ei), the number of the involved links is (-1) jk-1and the length of each link is kd; thus1 k 1Rk < -k Ink =- Ink = RTherefore-E opt(R)kdPr(ei) - 2e optThus, by Eqs. 7 and 10,-E (R)kdP(uk u) < 2m e optTherefore, by Eqs. 5, 6, and 11,Pr(Ck) < e - (R)n + 2m e opt124{ei}.(10)(11)(12)(IX. PROCESSING AND TRANSMISSION OF INFORMATION)Pr(Ck) < 2m eE(R)kdbecause, as we shall show, by Eq. 21, E(R) -< E opt(R).optPr(Ck - 1 ) < 2 me - E (R)( k - 1 )dThe average number of computations, by Eqs. 4 and 14, is therefore bounded bymN < 2m 2 p ke-E(R)(k-l)dk=lNow,R = 1 (k-1)d(k-l) dThus ek-1 = (k-1)dR. Thereforem2m 21 e(k-1)d[R-E(R)].k=Thus(14)(15)(16)Let Rcomp be the solution ofR = E(R)Then, for all rates below Rcomp, R - E(R) < 0, andN 2m2 1 (18)1_ e[R-E(R)]d .The average amount of computations is therefore an algebraic function of m for allrates below Rcompc. Evaluation of RcompFano has shown thatandE op t (R ) = E op t ( 0 ) - R;opt optopt optfor R < Ritcritfor R .< R < C.critLet us set D o so as to make E (R) of Eq. 6 equal toE(R) = Eopt(0) - Rcomp.125(13)(17)(19)(20)(21)(IX. PROCESSING AND TRANSMISSION OF INFORMATION)Thus, by Eqs. 12 and 21,R =E (O)-Rcomp opt comporR E (0). (22)comp 2 optAlso,E(R) =E (O)-R - E (0). (23)opt comp 2 optThus1N < 2m 2S- exp d R- Eopt(0)E opt(0) is the zero-rate exponent of the upper bound on the average probability of errorPe of the same channel, when the optimal decoding scheme is used. 52. The Average Probability of ErrorLet u be the transmitted code word. Let v be the corresponding channel outputvector. Let u' be one of the M(t-1) code words which starts with an information letterother than that of u. The probability of error is bounded byPe < Pr(D(u, v) Do) + M(-)Pr(D(u' , v)>Do;D(u, v) >D 0 ) (24)ThusP e Pr(D(u,v)<Do) + e n R Pr(D(u',v) >D ;D(u,v)>D ) (25)orPe < P (D(u,v)Do) + en R Pr(D(u' , v) >D;D(u, v) > D ) (26)LetP(v/u)D(u, v) = In (v)P( )P(v/u')D(u', v) = In (v)wherenP(v/u) = - P(yi/xi)i=l 1126(IX. PROCESSING AND TRANSMISSION OF INFORMATION)andnP(v) = -[ P(Yi )i=lP(Yi/xi) is the probability that the output is yi, given that the input symbol x i wastransmitted.P(xi ) P(Yi/Xi) = P(yi)X.ThusD(u, v)=i=lwheredi(xi, yi) = P(yi/xi)n P(yi)Thus by the use of the Chernoff Bound,Pr(D(u, v) < Do) < e n (G ( s )-s '(s));Pr(D(u', v) -< D )-< e n ( y(t)-ty'(t)).;s < 0t 0whereand(s) = P(x) P(y/x) esd(x, y)xyy(t) = P(x) P(x') P(y/x) es d( x 'Iy)xxyp(s) = P(x) P(y/x) l + s p(y)-sxyy(t) = P(x') P(y)l-t P(y/x')tx'y= P(x) P(y)l-t P(y/x)txy127di(xi' Yi),'(IX. PROCESSING AND TRANSMISSION OF INFORMATION)Also,D'(s) = y'(t) noIf we let t = 1 + s, it can be shown that y'(t) = 9'(t) (also y(t) = (s)). ThusP (D(u, v) Do) - en((s) - sF'(s))P (D(u', v) aD ) < e n ( ( s) - + s ) [ '( s ) ) 'whereD1(s) -nf(s) = ) P(x) P(y/x)1 + s p(y)-.xyThusP < en(4(s)-s '(s))+ en(R+[(s)-(l+s)'(s))eNow, by Eqs. 6 and 23, D o is set so as to getPr(D(u, v) < D )< en(4(s )- s - '(s)) = -n(R),whereE(R) = Eopt();and, as shown by Fano,for all R < Rcomp= -n xP(x) P(Y/x) 2y xWe shall now prove that, once D is set so as to make -i(s) +(28a)(28b)s'(s) = E(R) 1 Eopt(0),SC(S) :~E(R)as in Eq. 28, then1-l(s) + (i+s) ['(s) > 2 E opt(0).PROOF: The minimum of {(2s+l)p'(s)-2i(s)} occurs at that s for whicha s [(2s+1) '(s)-2z(s)] = 0(2s+l) '"(s ) + 2F'(s) - 2l'(s) = 01Thus s = - 12(29)128(27)(28)E opt(0) = E opt(R) R=0(IX. PROCESSING AND TRANSMISSION OF INFORMATION)Also,[(1+2s)~(s)-2[(s)] s=-1/2 0 2 , -(- ,since i" l(-) is the variance (see Fano5 )a minimum point. Thus[(2s+1)i'(s)-2[i(s) > -24 -.of a random variable.1 .Thus s = -. is indeed(30)Now- = n P (x) (y)/2 P(y/x)1/2xyand therefore4 1 = In P(x) P(y)12xywhere f(y) = P(x) P(y/x) 1/ 2.-J/2 p(y/x)l/2By the Schwarz inequality,{ f(y) P(y)1/2 2y< f(y )2 1p(y )Y YThusysince P(y) = 1.P(y) 2 7 P(y)y2yYTherefore, by Eq. 28b,2 (- In P(x)y xP(y/x)1/2 =-E opt(0).optThus -2z(-1) > Eopt(O), and therefore, by Eq. 30,129Thus224 In If(y) P(y)1/y(IX. PROCESSING AND TRANSMISSION OF INFORMATION)-[ E(s)-s '(s)i'(s)+(s) +s) (s)] > opt(0). (31)But, by Eq. 28,1-E(R) = t(s) - s4'(s) 2 Eopt(0).Therefore, by Eq. 31,4(s) - (l+s) 4'(s) > -- Eopt(0) Q.E.D. (32)Thus, by Eqs. 27, 28, and 32,P < e opt(0)n + en - Eopt(0)en E (0) n(R-R p )P 2 e n(R 2 Eopt = n(R-R compen(R-RcompP <2e compeIf Do is set so as to minimize the probability of error by making R+ [(s) - (1+s)['(s) =L(s)- s '(s) = E(R), Shannon 7 has shown that-P < 2 nE(R)P <2eewhere E(R) > 0 for R < C, and E(0) >Z E (0). However, R is then lower-boundedopt comp1 1by R > -E(0) > E (0).comp -2 E() >4 Ep t(O).This research was supported in part by a fellowship from the Government of Israel.J. ZivReferences1. J. M. Wozencraft and B. Reiffen, Sequential Decoding (The M.I.T. Press,Cambridge, Mass., and John Wiley and Sons, Inc., New York, 1961).2. R. G. Gallager, Low Density Parity Check Codes, Sc.D. Thesis, Departmentof Electrical Engineering, M.I.T., September 1960.3. B. Reiffen, Sequential Encoding and Decoding for the Discrete MemorylessChannel, Technical Report 374, Research Laboratory of Electronics, M.I.T.; TechnicalReport 231, Lincoln Laboratory, M.I.T., August 12, 1960, p. 29.4. Ibid., see Chapter 2.5. R. M. Fano, Transmission of Information (The M.I.T. Press, Cambridge,Mass., and John Wiley and Sons, Inc., New York, 1961), see Chapter 9.6. Ibid., see Chapter 8; B. Reiffen, o.F cit., see Appendix B.7. C. E. Shannon, Error Probability Bounds for Noisy Channels (unpublishedreport, 1959).130

Processing and Transmission of Information

http://dspace.mit.edu/bitstream/1721.1/53619/1/RLE_QPR_063_IX.pdf

Processing and Transmission of Information

Abstract

Similar works

Full text

Available Versions

DSpace@MIT