Stronger Connections Between Circuit Analysis and Circuit Lower Bounds, via PCPs of Proximity

We considerably sharpen the known connections between circuit-analysis algorithms and circuit lower bounds, show intriguing equivalences between the analysis of weak circuits and (apparently difficult) circuits, and provide strong new lower bounds for approximately computing Boolean functions with depth-two neural networks and related models. - We develop approaches to proving THR o THR lower bounds (a notorious open problem), by connecting algorithmic analysis of THR o THR to the provably weaker circuit classes THR o MAJ and MAJ o MAJ, where exponential lower bounds have long been known. More precisely, we show equivalences between algorithmic analysis of THR o THR and these weaker classes. The epsilon-error CAPP problem asks to approximate the acceptance probability of a given circuit to within additive error epsilon; it is the "canonical" derandomization problem. We show: - There is a non-trivial (2^n/n^{omega(1)} time) 1/poly(n)-error CAPP algorithm for poly(n)-size THR o THR circuits if and only if there is such an algorithm for poly(n)-size MAJ o MAJ. - There is a delta > 0 and a non-trivial SAT (delta-error CAPP) algorithm for poly(n)-size THR o THR circuits if and only if there is such an algorithm for poly(n)-size THR o MAJ. Similar results hold for depth-d linear threshold circuits and depth-d MAJORITY circuits. These equivalences are proved via new simulations of THR circuits by circuits with MAJ gates. - We strengthen the connection between non-trivial derandomization (non-trivial CAPP algorithms) for a circuit class C, and circuit lower bounds against C. Previously, [Ben-Sasson and Viola, ICALP 2014] (following [Williams, STOC 2010]) showed that for any polynomial-size class C closed under projections, non-trivial (2^{n}/n^{omega(1)} time) CAPP for OR_{poly(n)} o AND_{3} o C yields NEXP does not have C circuits. We apply Probabilistic Checkable Proofs of Proximity in a new way to show it would suffice to have a non-trivial CAPP algorithm for either XOR_2 o C, AND_2 o C or OR_2 o C. - A direct corollary of the first two bullets is that NEXP does not have THR o THR circuits would follow from either: - a non-trivial delta-error CAPP (or SAT) algorithm for poly(n)-size THR o MAJ circuits, or - a non-trivial 1/poly(n)-error CAPP algorithm for poly(n)-size MAJ o MAJ circuits. - Applying the above machinery, we extend lower bounds for depth-two neural networks and related models [R. Williams, CCC 2018] to weak approximate computations of Boolean functions. For example, for arbitrarily small epsilon > 0, we prove there are Boolean functions f computable in nondeterministic n^{log n} time such that (for infinitely many n) every polynomial-size depth-two neural network N on n inputs (with sign or ReLU activation) must satisfy max_{x in {0,1}^n}|N(x)-f(x)|>1/2-epsilon. That is, short linear combinations of ReLU gates fail miserably at computing f to within close precision. Similar results are proved for linear combinations of ACC o THR circuits, and linear combinations of low-degree F_p polynomials. These results constitute further progress towards THR o THR lower bounds

Stronger connections between circuit analysis and circuit lower bounds, via PCPs of proximity

© Lijie Chen and R. Ryan Williams; licensed under Creative Commons License CC-BY 34th Computational Complexity Conference (CCC 2019). We considerably sharpen the known connections between circuit-analysis algorithms and circuit lower bounds, show intriguing equivalences between the analysis of weak circuits and (apparently difficult) circuits, and provide strong new lower bounds for approximately computing Boolean functions with depth-two neural networks and related models. We develop approaches to proving THR ◦ THR lower bounds (a notorious open problem), by connecting algorithmic analysis of THR ◦ THR to the provably weaker circuit classes THR ◦ MAJ and MAJ ◦ MAJ, where exponential lower bounds have long been known. More precisely, we show equivalences between algorithmic analysis of THR ◦ THR and these weaker classes. The ε-error CAPP problem asks to approximate the acceptance probability of a given circuit to within additive error ε; it is the “canonical” derandomization problem. We show: There is a non-trivial (2n/nω(1) time) 1/poly(n)-error CAPP algorithm for poly(n)-size THR ◦ THR circuits if and only if there is such an algorithm for poly(n)-size MAJ ◦ MAJ. There is a δ > 0 and a non-trivial SAT (δ-error CAPP) algorithm for poly(n)-size THR ◦ THR circuits if and only if there is such an algorithm for poly(n)-size THR ◦ MAJ. Similar results hold for depth-d linear threshold circuits and depth-d MAJORITY circuits. These equivalences are proved via new simulations of THR circuits by circuits with MAJ gates. We strengthen the connection between non-trivial derandomization (non-trivial CAPP algorithms) for a circuit class C, and circuit lower bounds against C. Previously, [Ben-Sasson and Viola, ICALP 2014] (following [Williams, STOC 2010]) showed that for any polynomial-size class C closed under projections, non-trivial (2n/nω(1) time) CAPP for ORpoly(n) ◦ AND3 ◦ C yields NEXP 6⊂ C. We apply Probabilistic Checkable Proofs of Proximity in a new way to show it would suffice to have a non-trivial CAPP algorithm for either 2 ◦ C, AND2 ◦ C or OR2 ◦ C. A direct corollary of the first two bullets is that NEXP 6⊂ THR ◦ THR would follow from either: a non-trivial δ-error CAPP (or SAT) algorithm for poly(n)-size THR ◦ MAJ circuits, or a non-trivial 1/poly(n)-error CAPP algorithm for poly(n)-size MAJ ◦ MAJ circuits. Applying the above machinery, we extend lower bounds for depth-two neural networks and related models [R. Williams, CCC 2018] to weak approximate computations of Boolean functions. For example, for arbitrarily small ε > 0, we prove there are Boolean functions f computable in nondeterministic nlog n time such that (for infinitely many n) every polynomial-size depth-two neural network N on n inputs (with sign or ReLU activation) must satisfy maxx∈{0,1}n |N(x) − f(x)| > 1/2 − ε. That is, short linear combinations of ReLU gates fail miserably at computing f to within close precision. Similar results are proved for linear combinations of ACC ◦ THR circuits, and linear combinations of low-degree Fp polynomials. These results constitute further progress towards THR ◦ THR lower bounds