11 research outputs found
Super-Linear Gate and Super-Quadratic Wire Lower Bounds for Depth-Two and Depth-Three Threshold Circuits
In order to formally understand the power of neural computing, we first need
to crack the frontier of threshold circuits with two and three layers, a regime
that has been surprisingly intractable to analyze. We prove the first
super-linear gate lower bounds and the first super-quadratic wire lower bounds
for depth-two linear threshold circuits with arbitrary weights, and depth-three
majority circuits computing an explicit function.
We prove that for all , the
linear-time computable Andreev's function cannot be computed on a
-fraction of -bit inputs by depth-two linear threshold
circuits of gates, nor can it be computed with
wires. This establishes an average-case
``size hierarchy'' for threshold circuits, as Andreev's function is computable
by uniform depth-two circuits of linear threshold gates, and by
uniform depth-three circuits of majority gates.
We present a new function in based on small-biased sets, which
we prove cannot be computed by a majority vote of depth-two linear threshold
circuits with gates, nor with
wires.
We give tight average-case (gate and wire) complexity results for
computing PARITY with depth-two threshold circuits; the answer turns out to be
the same as for depth-two majority circuits.
The key is a new random restriction lemma for linear threshold functions. Our
main analytical tool is the Littlewood-Offord Lemma from additive
combinatorics
Circuit Complexity of Visual Search
We study computational hardness of feature and conjunction search through the
lens of circuit complexity. Let (resp., ) be Boolean variables each of which takes the value one if and only if a
neuron at place detects a feature (resp., another feature). We then simply
formulate the feature and conjunction search as Boolean functions and , respectively. We employ a threshold circuit or a discretized
circuit (such as a sigmoid circuit or a ReLU circuit with discretization) as
our models of neural networks, and consider the following four computational
resources: [i] the number of neurons (size), [ii] the number of levels (depth),
[iii] the number of active neurons outputting non-zero values (energy), and
[iv] synaptic weight resolution (weight).
We first prove that any threshold circuit of size , depth , energy
and weight satisfies ,
where is the rank of the communication matrix of a
-variable Boolean function that computes. Since has rank
, we have . Thus, an exponential
lower bound on the size of even sublinear-depth threshold circuits exists if
the energy and weight are sufficiently small. Since is computable
independently of , our result suggests that computational capacity for the
feature and conjunction search are different. We also show that the inequality
is tight up to a constant factor if . We next show that a
similar inequality holds for any discretized circuit. Thus, if we regard the
number of gates outputting non-zero values as a measure for sparse activity,
our results suggest that larger depth helps neural networks to acquire sparse
activity
Quantified Derandomization of Linear Threshold Circuits
One of the prominent current challenges in complexity theory is the attempt
to prove lower bounds for , the class of constant-depth, polynomial-size
circuits with majority gates. Relying on the results of Williams (2013), an
appealing approach to prove such lower bounds is to construct a non-trivial
derandomization algorithm for . In this work we take a first step towards
the latter goal, by proving the first positive results regarding the
derandomization of circuits of depth .
Our first main result is a quantified derandomization algorithm for
circuits with a super-linear number of wires. Specifically, we construct an
algorithm that gets as input a circuit over input bits with
depth and wires, runs in almost-polynomial-time, and
distinguishes between the case that rejects at most inputs
and the case that accepts at most inputs. In fact, our
algorithm works even when the circuit is a linear threshold circuit, rather
than just a circuit (i.e., is a circuit with linear threshold gates,
which are stronger than majority gates).
Our second main result is that even a modest improvement of our quantified
derandomization algorithm would yield a non-trivial algorithm for standard
derandomization of all of , and would consequently imply that
. Specifically, if there exists a quantified
derandomization algorithm that gets as input a circuit with depth
and wires (rather than wires), runs in time at
most , and distinguishes between the case that rejects at
most inputs and the case that accepts at most
inputs, then there exists an algorithm with running time
for standard derandomization of .Comment: Changes in this revision: An additional result (a PRG for quantified
derandomization of depth-2 LTF circuits); rewrite of some of the exposition;
minor correction