It is shown that if formulas are used to compute Boolean functions in the presence of randomly occurring failures then: (1) there is a limit strictly less than 1/2 to the failure probability per gate that can be tolerated, and (2) formulas that tolerate failures must be deeper (and, therefore, compute more slowly) than those that do not. The heart of the proof is an information-theoretic argument that deals with computation and errors in very general terms. The strength of this argument is that it applies with equal ease no matter what types of gate are available. Its weaknesses is that it does not seem to predict quantitatively the limiting value of the failure probability or the ratio by which computation proceeds more slowly in the presence of failures

Pippenger, Nicholas

English

Scholarship@Claremont

Claremont CollegesScholarship @ ClaremontAll HMC Faculty Publications and Research HMC Faculty Scholarship1-1-1988Reliable Computation by Formulas in the Presenceof NoiseNicholas PippengerHarvey Mudd CollegeThis Article is brought to you for free and open access by the HMC Faculty Scholarship at Scholarship @ Claremont. It has been accepted for inclusionin All HMC Faculty Publications and Research by an authorized administrator of Scholarship @ Claremont. For more information, please contactscholarship@cuc.claremont.edu.Recommended CitationPippenger, N., "Reliable computation by formulas in the presence of noise," Information Theory, IEEE Transactions on , vol.34, no.2,pp.194,197, Mar 1988. doi: 10.1109/18.2628194 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 34, NO. 2, MARCH 1988 Reliable Computation by Formulas in the Presence of Noise Abstract-It is shown that if formulas are used to compute Boolean functions in the presence of randomly occurring failures (as has been suggested by von Neumann and others), then 1) there is a limit shictly less than 1/2 to the failure probability per gate that can be tolerated, and 2) formulas that tolerate failures must be deeper (and, therefore, compute more slowly) than those that do not. I. INTRODUCTION COMMON way of computing Boolean functions is A by means of Boolean networks or formulas. In 1952, von Neumann [4] showed how this method could be made to work with high probability even when each gate in the network or formula failed independently with some small probability. The main idea in von Neumann’s construction is to interleave layers of gates that do computation with layers that do correction in such a way that error probabil- ities are kept under control. Two striking features of von Neumann’s construction are 1) that it works only when the failure probability per gate is sufficiently small (in the case considered by von Neumann, it has to be less than 1/6, though in general it depends on the types of gates avail- able), and 2) that computation proceeds more slowly than in the absence of failures since for a given failure probabil- ity a fixed fraction of the layers need to be devoted to correction. (No effort was made by von Neumann to estimate this fraction for this method. Some analysis of what can be achieved by this method d l  be given in Section I11 for purposes of comparison.) The goal of this paper is to show that, at least in the case of formulas, both of these features are essential no matter what types of gates are available and no matter how they are interconnected. (A formula is simply a network in which the output of each gate is used as the input to at most one other gate.) The difficulty of proving this lies in the condition “no matter how they are interconnected.” To appreciate this, consider that a formula that works correctly with high probability in the presence of randomly occurring failures need not work at all in the absence of failures. For computation, as distinct from communica- tion, it is possible that noise may be useful as well as detrimental, as is illustrated by the apparent power of randomized algorithms. Manuscript received October 1, 1986. The author is with the IBM Almaden Research Laboratory K51-802, 650 Harry Road, San Jose, CA 95120-6099. IEEE Log Number 8819893. The heart of our proof is an information-theoretic argu- ment that deals with computation and errors in very general terms. The strength of this argument is that it applies with equal ease no matter what types of gates are available. Its weakness is that it does not seem to predict quantitatively the limiting value of the failure probability or the ratio by which computation proceeds more slowly in the presence of failures. (In the case considered by von Neumann it shows only that the failure probability per gate must be less than 1/3, rather than 1/6. A comparison of computation speeds will be given in Section 111.) It remains an open question whether these features are essential for networks as well as formulas. Our proof breaks down for networks because the independence re- quired at a key point in the proof need not hold. Most previous work (for example, [l], [2], and [5]) has dealt with networks rather than formulas and with size (which corre- sponds to cost) rather than depth (which corresponds to delay). Tomb Feder, a graduate student in the Computer Science Department at Stanford University, Stanford, CA, has recently extended the results of this paper from for- mulas to networks. 11. MAIN RESULT Suppose that among the types of gates available there exists one that computes a Boolean function f that de- pends essentially on k 2 2  arguments but none that de- pend essentially on more than k arguments. By d-fold composition of f with itself, we obtain a Boolean function fd that depends essentially on n = k d  arguments. A formula may be regarded as a k-ary tree in which the leaves are either Boolean constants (0 or 1) or references to arguments (xl, x t ,  . ) and in which the internal nodes are gates computing Boolean functions of k arguments. The depth of a formula is the number of gates on the longest path from an argument reference to the root; it reflects the delay between the time that the values of the arguments are presented and the time that the value of the function is produced. Clearly, fd is computed (in the absence of failures) by a formula of depth d.  We shall show that any formula that computes f d  with high prob- ability in the presence of failures must be significantly deeper. Theorem: Let F be a formula with the following prop- erty: if each leaf of F operates correctly, and if each gate . 0018-9448/88/0300-Ol94$01.00 01988 IEEE PIPPENGER: RELIABLE COMPUTATION BY FORMULAS of F independently operates correctly with probability 1- E, where E > 0, then F operates correctly (that is, produces the value of fd) with probability at least 1 - 6, where 6 <1/2. Let A = 1 + 6 log, 6 + (1 - 6 )  log, (1 - 6 )  > 0. If E < (k  - 1)/2k, then the depth c of F satisfies where c 2 R d - S  log k R =  log((1-2c)k) 1% (l/A 1 log((1-2c)k) . and S =  (Since R > 1, computation proceeds more slowly in the presence of failures.) If z 2 (k - 1)/2k, then where d i T  log A T =  log (1 - 2 4  . (Thus if c is too large, only relatively simple functions can be computed reliably in the presence of failures.) The proof is by way of information theory. For 11 m I n, let F, denote the following formula. Since fd depends essentially on x,, Boolean con- stants c , ;  . -, c,- ,, c,+,; - -, c ,  exist such that fd(cl; . + ,  c,-,, x,, c , + ~ , -  . e ,  c,) is either x, or its com- plement x,. Substitute these constants for all references to arguments other than x, in F to obtain F,. Let X be a Boolean random variable that assumes the values 0 and 1 with equal probabilities. For any formula G, let Y ( G )  be the Boolean random variable produced by G when X is substituted for all argument references in G, and each gate in G fails with probability E, independently of X and of all other gates. By assumption, either Pr(Y(F,) = X) 21 - 6 or Pr(Y(F,) = X) 21 - 6. In either case, we have by Fano’s lemma - Z(Y(F,); X) 2 A .  (1) (See [3, theorem 4.11 for Fano’s lemma.) If G is a formula, let @(G, E )  be the generating function for the paths from argument references to the root in G, classified according to the number of gates they contain. That is, if L ( P )  denotes the number of gates on the path P, then @(G, () = P Inequality (1) gives a lower bound to Z(Y(F,); X) in terms of 6. To exploit it we shall prove an upper bound in terms of E, namely, Z(Y(F,);  x) I@(Fm,1-2E).  (2) For now, let us see how (1) and (2) imply the Theorem. Combining (1) and (2) yields @(Fm,1-2e) >A,  and summing over m in the range 1 I m I n yields @ ( F , 1 - 2 ~ )  2 n A  since @ ( F ,  t )  = C ‘ ( F m ,  0-  @ ( F , l / k )  SI. @ ( G , l / k )  21 l s r n s n  We shall need the inequality To prove this we shall show that for any k-ary formula G. We shall proceed by induction on the structure of G. If G is a constant leaf, @ ( G ,  5) = 0. If G is an argument reference leaf, @ ( G ,  5) = to  =l. For the inductive step, suppose that the root of G is a gate with subformulas G,;  - e ,  G,. Then @ ( G J ) = t  c @(G,,5). (8) l S j S k  Thus @(G, l /k )  = ( l / k )  @(G, , l /k) .  Applying the inductive hypothesis to each term on the right completes the proof of (7). (Inequality (7) is equiv- alent in content to the Kraft-Szilard inequality; see [3, lemma AI] . )  From (5) and (6) it follows that for some m in the range 1 I m I n we have 1 S j S k  @ ( F , , l / k )  s l / n .  (9) Suppose first that c < ( k  - 1)/2k. Setting X = log,/, (1 - 2z), we have 0 < X < 1. Holder’s inequality (see [3, lemma C.41) reads C A P B P S  (CA\P))”(  X B ; y - ) ” ) ) l - * .  P P P Putting A, = (1 - 2 ~ ) ~ ( ~ )  and B p= 1, and letting P run over paths from argument references to the root in F, yields @( F ,  1 - 2 4  I @ (  F , l / k ) ) ” @ (  F, l ) l -  )”. Bounding @(F,1-2c) below by (4) and bounding @( F, l / k )  above by (6) yields @ ( F , l )  2 (nA)”(’-)”). (10) The left side, @( F, l), is simply the number of argument reference leaves in F; if F has depth c, this is at most k“. On the other hand, n = kd.  Substituting these relations in (10) and taking logarithms to base k yields the first part of the theorem. Now suppose that 6 2 ( k  - 1)/2k. Setting p = log, - 2c ( l /k) ,  we have 0 < p I 1. Jensen’s inequality (see [3, lemma C.61) reads P 196 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 34, NO. 2, MARCH 1988 Putting A ,  = ( l / k ) L ( p )  and letting P run over all paths from argument reference leaves to the root in F, yields cp ( F, , l / k  ) 2 cp ( F, , 1 - 2 4  p .  Bounding @(F,,l /k) above by (9) and @ ( F m , l - 2 e )  be- low by (3) yields l / n  2 h p .  Substituting n = k d  and taking logarithms to base k yields the second part of the theorem. It remains to prove (2). To do this we shall show that I ( Y ( G ) ;  X )  1 @ ( G , 1 - 2 € )  (11) for any formula G. We shall proceed by induction on the structure of G. If G is a constant leaf, I (Y(G) ;  X )  and (P(G,1-2c)  both vanish. If G is an argument reference leaf, then I ( Y ( G ) ;  X ) = H ( X ) a n d O ( G , 1 - 2 ~ ) = ( 1 - 2 e ) ~  both equal unity. For the inductive step, suppose that the root of G is a gate that computes a Boolean function g of the values produced by the subformulas G,; e ,  G,. Since G is a formula, the random variables Y(G,) , .  e ,  Y(G,)  are conditionally independent given the value of X (since in this case they depend only on the failures, which are independent in the disjoint subformulas G1, -  . 0 ,  Gk) .  Thus we have q y ( G J , - , y ( G &  x) I c I ( Y ( G , ) ;  x). I S J l k  (See [3, lemma 6.31 for this inequality.) By the data processing theorem, I (  g (  Y (G1) , .  . ., Y ( G k ) ) ;  X, I I (  Y ( G l ) , .  * * 9 y(  G k ) ;  X, c I (Y(G, ) ;  x). 1 S j l k  (See [3, theorem 4.21 for this theorem.) Now Y ( G )  = g ( Y ( G , ) , - .  , Y ( G k ) ) +  W, where “+” deaotes the sum modulo 2 (“exclusive or”), and W is a Boolean random variable that assumes the value 1 with probability e and the value 0 with probability 1 - e ,  independently of X and g (  Y(G,) ;  . . , Y(G,)).  By the following lemma we have I ( Y ( G ) ;  X )  (1-26)1( g ( Y ( G i ) , * .  * ,  Y ( G , ) ) ;  X) I ( 1 - 2 ~ )  I ( Y ( G , ) ;  X ) .  l l j l k  Applying the inductive hypothesis to each term on the right and using (8) completes the proof of (11). It remains to prove the following lemma. Lemma: Let X and V be Boolean random variables, and let Y = V + W, where W is a Boolean random variable that assumes the value 1 with probability e and the value 0 with probability 1 - c, independently of X and V. Then I ( Y ;  X )  I (1--2e)Z(V; X). Proof: Let U be a Boolean random variable that assumes the values 0 and 1 with equal probabilities, inde- pendently of X and V. Let W‘ be a Boolean random variable that assumes the value 1 with probability 2r and the value 0 with probability 1 - 2 e ,  independently of X ,  V, and / J .  Let the random variable Y’ equal U if W’=l  and equal V if W’ = 0. Then X ,  V,  and Y’ have the same joint distribution as X, V, and Y,  so it will suffice to show that Z(Y’; x) I ( 1 - 2 e ) z ( v ;  x). This is done by an easy calculation: I (  Y’;  x) I I ( (  Y’, w); x) = I (  Y’; X l W )  + I (  w’; x) =Pr(W’=O)I(V; X )  + P r ( W = I ) I ( U ;  x ) + I ( w ’ ;  x ) .  The first term is (1 -2e)Z(V; X ) ,  and the remaining two terms vanish since U and W’ are independent of X .  111. CONCLUSION We have shown that if k-ary formulas are to compute arbitrarily complex functions with error probability bounded below 1/2, the failure probability per gate must be bounded below (k - 1)/2k < 1/2. This may be surpris- ing considering that a binary symmetric channel (which corresponds to a unary identity gate in our model) with failure probability e has capacity 1 + e log, e + (1 - e)log, (1 - e), which remains positive for E < 1/2. The reason for this discrepancy is evidently that when discuss- ing reliable communication in the presence of noise, the definition of capacity allows unlimited Computation without errors by the encoder and decoder; for reliable computation in the presence of noise, however, any encod- ing or decoding must be done by components that are themselves subject to failures. Only when the errors intro- duced by a gate can be offset by the computation per- formed by that gate can reliable computation in the pres- ence of noise be done. Finally, we shall give a quantitative comparison of the negative results in this paper with the positive results implicit in von Neumann’s paper. The case considered throughout the greater part of von Neumann’s paper is that in which errors are corrected by three-way voting using a three-argument majority gate. If the inputs to such a gate are independently incorrect with probability 5 and if the gate fails with probability e, then the output is incorrect with probability f( 0, where f(<) = c + ( 1 - 2 ~ ) ( 3 [ ~  - 2 t 3 ) .  The error probability 6 must be greater than the smallest solution q ( e )  of the equation f([) = 5. We have q ( e )  < 1/2 if and only if e < 1/6. For k = 3, however, our result only gives c <1/3, and it remains an open question whether any scheme can achieve 6 < 1/2 for 1/6 I e < 1/3. To compare bounds on the speed of computation, we must determine the fraction of layers that must be devoted to correction. This fraction depends on what is done by the layers devoted to computation. The simplest assumption is that the gates in the layers devoted to computation are three-argument parity gates (a discussion of the general case will be given later). If the inputs to such a gate are independently incorrect with probability ,$ and the gate fails with probability e, then the output is incorrect with PIPPENGER: RELIABLE COMPUTATION BY FORMULAS 197 probability g( t) ,  where g ( [ )  = E  + (1 -2~) (3 (  -6 t2  +4t3) .  Let us consider first the case E + 0. The order of magni- tude of 4(c) is E. The effect of applying f is roughly to square its argument (it also triples the result and adds E, but these effects are less important). Thus a correction should be applied when the error probability has order of magnitude d/’. The effect of applying g is roughly to triple its argument (again, it also adds c,  but this effect is less important). Thus the number of layers of computation that can be put between one correction layer (after which the error probability has order of magnitude E) and another (before which it has order of magnitude d/’) is asymptotic to log, ( c ~ / ~ / E )  = (1/2)log, ( l /c ) .  Thus a fraction asymp- totic to 2/10g,(l/~) of the layers must be devoted to correction. Our result says only that this fraction must be at least asymptotic to 2r/ln3, which vanishes much more rapidly as E + 0. Let us now consider the case E + 1/6. To do this we write E =1/6- 6 and let 6 -+ 0. In this case q ( ~ )  -+1/2, so we write A([) =1/2- [ and find that h (4 (~ ) )  has order of magnitude 6112. The effect of applying h 0 g o  h is roughly to cube its argument, so after a computation layer the error probability has order of magnitude $,I2. The effect of applying h 0 f 0 h is roughly to multiply its argu- ment by f’(1/2) = 1 + 39. Thus the number of correction layers that need to be put between one computation layer (after which the error probability has order of magnitude $,I2) and another (before which it has order of magnitude a l l 2 )  is asymptotic to log,+,s (81/2/63/2)  - (1/36) ln(l/8). Thus only a fraction asymptotic to 39/ln(1/9) of the layers can be devoted to computation. Our result says only that at most a fraction log,2 of the layers can be devoted to computation. The analysis just given can be extended to cases in which gates other than parity gates appear in the layers devoted to computation, but some care is needed. For parity gates, increasing the error probability of an input increases the error probability of the output, but in general this need not be the case. In the absence of such mono- tonicity, one must maintain lower bounds to error prob- ability as well as upper bounds. When this is done, how- ever, it turns out that parity constitutes the worst case, even when all three-argument gates are available. In other special cases (for example, when only majority and minor- ity gates are available) more favorable bounds can be obtained. It remains an open problem to obtain stronger negative results than those given here. In particular, it would be interesting to prove either 1) that reliable computation of arbitrarily complex functions cannot be done by three- argument gates if e zl/6, or 2) that the fraction of layers that must be devoted to correction must have order of magnitude l/ln(l/c) for small E. REFERENCES R. L. Dobrushin and S. I. Ortyukov, “Upper bound for the re- dundancy of self-correcting arrangements of unreliable functional elements,”, Prob. Inform. Transm., vol. 13, pp. 203-218, 1977. -, “Lower bound for the redundancy of self-correcting arrange- ments of unreliable functional elements,” Proh. Inform. Transm., F. Jelinek, Probabilistic Information Theory. New York: McGraw- Hill, 1968. J. von Neumann, “Probabilistic logics and the synthesis of reliable organisms from unreliable components,” in Automata Studies, C. E. Shannon and J. McCarthy, Eds. Princeton, NJ: Princeton Univ. Press, 1956, pp. 43-98. N. Pippenger, “On networks of noisy gates,” in Proc. IEEE Symp. Found. of Comp. Sci., vol. 26, 1985, pp. 30-38. V O ~ .  13, pp. 59-65, 1977. 

Reliable Computation by Formulas in the Presence of Noise

https://scholarship.claremont.edu/cgi/viewcontent.cgi?article=1574&amp;context=hmc_fac_pub

Reliable Computation by Formulas in the Presence of Noise

Abstract

Similar works

Full text

Available Versions

Scholarship@Claremont