One fundamental problem in deep learning is understanding the outstanding
performance of deep Neural Networks (NNs) in practice. One explanation for the
superiority of NNs is that they can realize a large class of complicated
functions, i.e., they have powerful expressivity. The expressivity of a ReLU NN
can be quantified by the maximal number of linear regions it can separate its
input space into. In this paper, we provide several mathematical results needed
for studying the linear regions of CNNs, and use them to derive the maximal and
average numbers of linear regions for one-layer ReLU CNNs. Furthermore, we
obtain upper and lower bounds for the number of linear regions of multi-layer
ReLU CNNs. Our results suggest that deeper CNNs have more powerful expressivity
than their shallow counterparts, while CNNs have more expressivity than
fully-connected NNs per parameter.Comment: International Conference on Machine Learning (ICML) 202