539 research outputs found
Towards Effective Low-bitwidth Convolutional Neural Networks
This paper tackles the problem of training a deep convolutional neural
network with both low-precision weights and low-bitwidth activations.
Optimizing a low-precision network is very challenging since the training
process can easily get trapped in a poor local minima, which results in
substantial accuracy loss. To mitigate this problem, we propose three
simple-yet-effective approaches to improve the network training. First, we
propose to use a two-stage optimization strategy to progressively find good
local minima. Specifically, we propose to first optimize a net with quantized
weights and then quantized activations. This is in contrast to the traditional
methods which optimize them simultaneously. Second, following a similar spirit
of the first method, we propose another progressive optimization approach which
progressively decreases the bit-width from high-precision to low-precision
during the course of training. Third, we adopt a novel learning scheme to
jointly train a full-precision model alongside the low-precision one. By doing
so, the full-precision model provides hints to guide the low-precision model
training. Extensive experiments on various datasets ( i.e., CIFAR-100 and
ImageNet) show the effectiveness of the proposed methods. To highlight, using
our methods to train a 4-bit precision network leads to no performance decrease
in comparison with its full-precision counterpart with standard network
architectures ( i.e., AlexNet and ResNet-50).Comment: 11 page
Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networks
Quantized Neural Networks (QNNs), which use low bitwidth numbers for
representing parameters and performing computations, have been proposed to
reduce the computation complexity, storage size and memory usage. In QNNs,
parameters and activations are uniformly quantized, such that the
multiplications and additions can be accelerated by bitwise operations.
However, distributions of parameters in Neural Networks are often imbalanced,
such that the uniform quantization determined from extremal values may under
utilize available bitwidth. In this paper, we propose a novel quantization
method that can ensure the balance of distributions of quantized values. Our
method first recursively partitions the parameters by percentiles into balanced
bins, and then applies uniform quantization. We also introduce computationally
cheaper approximations of percentiles to reduce the computation overhead
introduced. Overall, our method improves the prediction accuracies of QNNs
without introducing extra computation during inference, has negligible impact
on training speed, and is applicable to both Convolutional Neural Networks and
Recurrent Neural Networks. Experiments on standard datasets including ImageNet
and Penn Treebank confirm the effectiveness of our method. On ImageNet, the
top-5 error rate of our 4-bit quantized GoogLeNet model is 12.7\%, which is
superior to the state-of-the-arts of QNNs
- …