We report on two types of results. The first is a study of the rate of decay of information carried by a signal which is being propagated over a noisy channel. The second is a series of lower bounds on the depth, size, and component reliability of noisy logic circuits which are required to compute some function reliably. The arguments used for the circuit results are information-theoretic, and in particular, the signal decay result is essential to the depth lower bound. Our first result can be viewed as a quantified version of the data processing lemma, for the case of Boolean random variables

Evans, William S.

Schulman, Leonard J.

English

int distribution for X and Y so that  I(X;Z)=I(X;Y ) is arbitrarily close to sin  2  `.  The previous theorem is a general result about mutual information. The remaining theorems concern the noisy circuit model of Von Neumann [7]. The signal decay theorem is useful in proving lower bounds on the structure of such circuits whose components (i.e. individual logic gates) fail with some probability. These results improve and simplify all previous lower bounds in this model.  Theorem 2 (Noisy Circuit Depth) Let f be a Boolean function which depends on n inputs. Let C be a circuit of depth c using gates with at most k inputs, where each gate fails independently with probability (1 \Gamma ¸)=2. Suppose C c

William S. Evans

Leonard J. Schulman

CiteSeerX

Information Theory and Noisy Computation

Caltech Authors - Main

Information Theory and Noisy Computation 
William S. Evans] 
Dept. of Computer Science, Univ. British Columbia 
We report on two types of results. The first is a study of 
the rate of decay of information carried by a signal which is 
being propagated over a noisy channel. The second is a series 
of lower bounds on the depth, size, and component reliability 
of noisy logic circuits which are required to compute some 
function reliably. The arguments used for the circuit results 
are information-theoretic, and in particular, the signal decay 
result is essential to  the depth lower bound. 
Our first result can be viewed as a quantified version of 
the da ta  processing lemma, for the case of Boolean random 
variables. 
Theorem 1 (Signal Decay) I j X ,  Y are Boolean r a n d o m  
varzables and Z zs t he  ou tpu t  of the channel  [ b 1 - 61 
o n  inpu t  Y t h e n  ~ < sin'@, where o zs the  angle zn the 
1 - a  a 
I ( X ; l ' I  - ~ . ,  
p lane  between the vectors (G, 6) and (4, J1-b) 
I t  is worth emphasizing that the bound holds regardless of 
the distribution on X and Y, and is a property of the chan- 
nel alone. The bound is tight in tha t  for any such channel, 
one can describe a joint distribution for X and Y so that 
I ( X ;  Z ) / I ( X ;  Y )  is arbitrarily close to sin2 B. 
The previous theorem is a general result about mutual in- 
formation. The remaining theorems concern the noisy circuit 
model of Von Neumann [7]. The signal decay theorem is use- 
ful in proving lower bounds on the structure of such circuits 
whose components (i.e. individual logic gates) fail with some 
probability. These results improve and simplify all previous 
lower bounds in this model. 
Theorem 2 (Noisy Circuit Depth) Let  f be a Boolean 
function which depends on  n inpu t s .  Le t  C be a circuit of 
depth c using gates w i th  at m o s t  k i npu t s ,  where each gate 
fai ls  independent ly  w i th  probability (1 - < ) / 2 .  Suppose C com- 
pu te s  the  f u n c t i o n  f correctly o n  all i npu t s  wzth probabzlity at 
least 1-6 where 6 < 112. Let  A = 1+6log6+(1-6)log(l-6).  
e If E 2  > l / k  t h e n  c 2 
e If t2 5 l / k  t h e n  n 5 l / A  
To prove this theorem, we analyze the mutual information 
between the input to the noisy circuit and its output. This 
information must be large since the circuit reliably computes 
the function f ;  yet, according to  the signal decay theorem, 
each noisy gate in the circuit, when viewed as a noisy channel, 
decreases information. Together, these observations imply the 
lower bound on circuit depth. This improves on the lower 
bounds of Pippenger [ 5 ]  and Feder [l]. 
A similar technique, using a different measure of correlation 
than mutual information, provides a lower bound on noisy 
circuit size. 
'Supported by a Canadian International Fellowship and NSF 
'Supported in part by an NSF Postdoctoral Fellowship. 
grant CCR 92-01092. wevansmcs .ubc. ca 
Leonard J.  Schulman2 
Dept. of Applied Mathematics, Weizmann Inst. Science 
College of Computing, Georgia Inst. Technology 
Theorem 3 (Noisy Circuit Size) Let  f be a Boolean f u n c -  
taon w i th  sensitzvity' s .  Let  C be a circuit using gates w i th  at 
m o s t  k i npu t s ,  where each gate fai ls  independent ly  w i th  proba- 
bzlity ( 1  - < ) / 2 .  Suppose C computes  the  j u n c t i o n  f correctly 
o n  all i npu t s  w i th  probability at least 1 - 6 where 6 < 112, t h e n  
the n u m b e r  of gates in C i s  at  least s10gs+2s'og('(1-26))  where k loe t 
Previously. GQ1 [ 3 ] ,  Reischuk and Schmeltz [6], and GBcs and 
G d  [2] proved an n ( s  log s) bound on reliable circuit size. Our 
improvement is in the bound's dependence on component re- 
liability. 
Finally. we establish a threshold for component reliability 
below which one cannot reliably compute all functions. 
Theorem 4 (Component Reliability) For k odd there ex- 
is ts  6 < 112 such  that  for  all Boolean func t ions  f there exists 
a fo rmula2(us ing  gates w i th  at m o s t  k i npu t s ,  where each gate 
fai ls  independent ly  w i th  probability e) which  compu tes  f cor- 
rectly o n  all i npu t s  w i th  probability at least 1 - 6 if  and only  
if  
1 2k--2 
This extends work done by Hajek and Weller [4], who 
showed the result for k = 3. 
REFERENCES 
T. Feder. Reliable computation by networks in the presence of 
noise. IEEE Transactions on Information Theory, 35(3):569- 
571, May 1989 
P. GQcs and A. Gal. Lower bounds for the complexity of reli- 
able Boolean circuits with noisy gates. IEEE Transactions on 
Information Theory, 40(2):579-583, March 1994. 
A.  Gal. Lower bcunds for the complexity of reliable Boolean 
circuits with noisy gates. In Proceedings of the 32nd Annual 
Symposium on  Foundations of Computer Science, pages 594- 
601, 1991. 
B. Hajek and T. Weller. On the maximum tolerable noise for 
reliable computation by formulas. IEEE Transactions on In- 
formation Theory, 37(2):388-391, March 1991. 
N. Pippenger. Reliable computation by formulas in the pres- 
34(2):194-197, March 1988. 
R. Reischuk and B. Schmeltz. Reliable computation with noisy 
circuits and decision trees - a general n log n lower bound. In 
Proceedings of the 32nd Annual Symposium o n  Foundations of 
Computer Science, pages 602-611, 1991. 
J .  von Neumann. Probabilistic logics and the synthesis of re- 
liable organisms from unreliable components. In C. E. Shan- 
non and J .  McCarthy, editors, Automata Studies, pages 43-98. 
Princeton University Press, 1956. 
ence of noise. IEEE Transactions on Information Theory ,  
'The sensitivity of a function is the maximum (over all inputs) of 
the number of bits in the input which. when changed individually, 
change the function value. 
schulman@cc.gatech.edu 'A formula is a circuit in which each gate has out-degree one 
456 


Lower bcunds for the complexity of reliable Boolean circuits with noisy gates.

On the maximum tolerable noise for reliable computation by formulas.

Reliable computation by formulas in the pres34(2):194-197,

Reliable computation by networks in the presence of noise.

Reliable computation with noisy circuits and decision trees - a general n log n lower bound.

http://authors.library.caltech.edu/29341/1/EVAisit95.pdf

Information Theory and Noisy Computation

Abstract

Similar works

Full text

Available Versions

CiteSeerX

Caltech Authors - Main