Multiple Threshold Neural Logic

We introduce a new Boolean computing element related to the Linear Threshold element, which is the Boolean version of the neuron. Instead of the sign function, it computes an arbitrary (with polynomialy many transitions) Boolean function of the weighted sum of its inputs. We call the new computing element an LT M element, which stands for Linear Threshold with Multiple transitions. The paper consists of the following main contributions related to our study of LTM circuits: (i) the creation of efficient designs of LTM circuits for the addition of a multiple number of integers and the product of two integers. In particular, we show how to compute the addition of m integers with a single layer of LT M elements. (ii) a proof that the area of the VLSI layout is reduced from O(n^2) in LT circuits to O(n) in LTM circuits, for n inputs symmetric Boolean functions, and (iii) the characterization of the computing power of LT M relative to LT circuits

Stronger Connections Between Circuit Analysis and Circuit Lower Bounds, via PCPs of Proximity

We considerably sharpen the known connections between circuit-analysis algorithms and circuit lower bounds, show intriguing equivalences between the analysis of weak circuits and (apparently difficult) circuits, and provide strong new lower bounds for approximately computing Boolean functions with depth-two neural networks and related models. - We develop approaches to proving THR o THR lower bounds (a notorious open problem), by connecting algorithmic analysis of THR o THR to the provably weaker circuit classes THR o MAJ and MAJ o MAJ, where exponential lower bounds have long been known. More precisely, we show equivalences between algorithmic analysis of THR o THR and these weaker classes. The epsilon-error CAPP problem asks to approximate the acceptance probability of a given circuit to within additive error epsilon; it is the "canonical" derandomization problem. We show: - There is a non-trivial (2^n/n^{omega(1)} time) 1/poly(n)-error CAPP algorithm for poly(n)-size THR o THR circuits if and only if there is such an algorithm for poly(n)-size MAJ o MAJ. - There is a delta > 0 and a non-trivial SAT (delta-error CAPP) algorithm for poly(n)-size THR o THR circuits if and only if there is such an algorithm for poly(n)-size THR o MAJ. Similar results hold for depth-d linear threshold circuits and depth-d MAJORITY circuits. These equivalences are proved via new simulations of THR circuits by circuits with MAJ gates. - We strengthen the connection between non-trivial derandomization (non-trivial CAPP algorithms) for a circuit class C, and circuit lower bounds against C. Previously, [Ben-Sasson and Viola, ICALP 2014] (following [Williams, STOC 2010]) showed that for any polynomial-size class C closed under projections, non-trivial (2^{n}/n^{omega(1)} time) CAPP for OR_{poly(n)} o AND_{3} o C yields NEXP does not have C circuits. We apply Probabilistic Checkable Proofs of Proximity in a new way to show it would suffice to have a non-trivial CAPP algorithm for either XOR_2 o C, AND_2 o C or OR_2 o C. - A direct corollary of the first two bullets is that NEXP does not have THR o THR circuits would follow from either: - a non-trivial delta-error CAPP (or SAT) algorithm for poly(n)-size THR o MAJ circuits, or - a non-trivial 1/poly(n)-error CAPP algorithm for poly(n)-size MAJ o MAJ circuits. - Applying the above machinery, we extend lower bounds for depth-two neural networks and related models [R. Williams, CCC 2018] to weak approximate computations of Boolean functions. For example, for arbitrarily small epsilon > 0, we prove there are Boolean functions f computable in nondeterministic n^{log n} time such that (for infinitely many n) every polynomial-size depth-two neural network N on n inputs (with sign or ReLU activation) must satisfy max_{x in {0,1}^n}|N(x)-f(x)|>1/2-epsilon. That is, short linear combinations of ReLU gates fail miserably at computing f to within close precision. Similar results are proved for linear combinations of ACC o THR circuits, and linear combinations of low-degree F_p polynomials. These results constitute further progress towards THR o THR lower bounds

A short list of Equalities induces large sign-rank

We exhibit a natural function Fn on n variables that can be computed by just a linear-size decision list of "Equalities," but whose sign-rank is 2Ω (n1/4). This yields the following two new unconditional complexity class separations. 1. Boolean circuit complexity. The function Fn can be computed by linear-size depth-two threshold formulas when the weights of the threshold gates are unrestricted (THR ∘ THR), but any THR ∘ MAJ circuit (the weights of the bottom threshold gates are polynomially bounded in n) computing Fn requires size 2Ω (n1/4). This provides the first separation between the Boolean circuit complexity classes THR ∘ MAJ and THR ∘ THR. While Amano and Maruoka [Proceedings of the 30th International Symposium on Mathematical Foundations of Computer Science, 2005, pp. 107-118] and Hansen and Podolskii [Proceedings of the 25th Annual IEEE Conference on Computational Complexity, 2010, pp. 270-279] emphasized that superpolynomial separations between the two classes remained a basic open problem, our separation is in fact exponential. In contrast, Goldmann, Håstad, and Razborov [Comput. Complexity, 2 (1992), pp. 277-300] showed more than twenty-five years ago that functions efficiently computable by MAJ ∘ THR circuits can also be efficiently computed by MAJ ∘ MAJ circuits. In view of this, it was not even clear if THR ∘ THR was significantly more powerful than THR ∘ MAJ until our work, and there was no candidate function identified for the potential separation. 2. Communication complexity. The function Fn (under the natural partition of the inputs) lies in the communication complexity class PMA. Since Fn has large sign-rank, this implies PMA ⊈ UPP, strongly resolving a recent open problem posed by Göös, Pitassi, and Watson [Comput. Complexity, 27 (2018), pp. 245-304]. In order to prove our main result, we view Fn as an XOR function and develop a technique to lower bound the sign-rank of such functions. This requires novel approximation-theoretic arguments against polynomials of unrestricted degree. Further, our work highlights for the first time the class "decision lists of exact thresholds" as a common frontier for making progress on longstanding open problems in threshold circuits and communication complexity

The chow parameters problem

In the 2nd Annual FOCS (1961), C. K. Chow proved that every Boolean threshold function is uniquely determined by its degree-0 and degree-1 Fourier coefficients. These numbers became known as the Chow Parameters. Providing an algorithmic version of Chow’s theorem — i.e., efficiently construct-ing a representation of a threshold function given its Chow Parameters — has remained open ever since. This problem has received significant study in the fields of circuit complexity [Elg60, Cho61, Der65, Win71], game theory and the design of voting systems [DS79, Lee03, TT06, APL07], and learning theory [BDJ+98, Gol06]. In this paper we effectively solve the problem, giving a randomized PTAS with the following behav-ior: Theorem: Given the Chow Parameters of a Boolean threshold function f over n bits and any con-stant > 0, the algorithm runs in time O(n2 log2 n) and with high probability outputs a representation of a threshold function f ′ which is -close to f. Along the way we prove several new results of independent interest about Boolean threshold func-tions. In addition to various structural results, these include the following new algorithmic results i