469 research outputs found
Bimodal Distributed Binarized Neural Networks
Binary Neural Networks (BNNs) are an extremely promising method to reduce
deep neural networks' complexity and power consumption massively. Binarization
techniques, however, suffer from ineligible performance degradation compared to
their full-precision counterparts.
Prior work mainly focused on strategies for sign function approximation
during forward and backward phases to reduce the quantization error during the
binarization process. In this work, we propose a Bi-Modal Distributed
binarization method (\methodname{}). That imposes bi-modal distribution of the
network weights by kurtosis regularization. The proposed method consists of a
training scheme that we call Weight Distribution Mimicking (WDM), which
efficiently imitates the full-precision network weight distribution to their
binary counterpart. Preserving this distribution during binarization-aware
training creates robust and informative binary feature maps and significantly
reduces the generalization error of the BNN. Extensive evaluations on CIFAR-10
and ImageNet demonstrate the superiority of our method over current
state-of-the-art schemes. Our source code, experimental settings, training
logs, and binary models are available at
\url{https://github.com/BlueAnon/BD-BNN}
- …