19 research outputs found
Layerwise Noise Maximisation to Train Low-Energy Deep Neural Networks
Deep neural networks (DNNs) depend on the storage of a large number of
parameters, which consumes an important portion of the energy used during
inference. This paper considers the case where the energy usage of memory
elements can be reduced at the cost of reduced reliability. A training
algorithm is proposed to optimize the reliability of the storage separately for
each layer of the network, while incurring a negligible complexity overhead
compared to a conventional stochastic gradient descent training. For an
exponential energy-reliability model, the proposed training approach can
decrease the memory energy consumption of a DNN with binary parameters by
3.3 at isoaccuracy, compared to a reliable implementation.Comment: To be presented at AICAS 202
Projection Convolutional Neural Networks for 1-bit CNNs via Discrete Back Propagation
The advancement of deep convolutional neural networks (DCNNs) has driven
significant improvement in the accuracy of recognition systems for many
computer vision tasks. However, their practical applications are often
restricted in resource-constrained environments. In this paper, we introduce
projection convolutional neural networks (PCNNs) with a discrete back
propagation via projection (DBPP) to improve the performance of binarized
neural networks (BNNs). The contributions of our paper include: 1) for the
first time, the projection function is exploited to efficiently solve the
discrete back propagation problem, which leads to a new highly compressed CNNs
(termed PCNNs); 2) by exploiting multiple projections, we learn a set of
diverse quantized kernels that compress the full-precision kernels in a more
efficient way than those proposed previously; 3) PCNNs achieve the best
classification performance compared to other state-of-the-art BNNs on the
ImageNet and CIFAR datasets
Binarized Neural Architecture Search
Neural architecture search (NAS) can have a significant impact in computer
vision by automatically designing optimal neural network architectures for
various tasks. A variant, binarized neural architecture search (BNAS), with a
search space of binarized convolutions, can produce extremely compressed
models. Unfortunately, this area remains largely unexplored. BNAS is more
challenging than NAS due to the learning inefficiency caused by optimization
requirements and the huge architecture space. To address these issues, we
introduce channel sampling and operation space reduction into a differentiable
NAS to significantly reduce the cost of searching. This is accomplished through
a performance-based strategy used to abandon less potential operations. Two
optimization methods for binarized neural networks are used to validate the
effectiveness of our BNAS. Extensive experiments demonstrate that the proposed
BNAS achieves a performance comparable to NAS on both CIFAR and ImageNet
databases. An accuracy of vs. is achieved on the CIFAR-10
dataset, but with a significantly compressed model, and a faster search
than the state-of-the-art PC-DARTS
Single-bit-per-weight deep convolutional neural networks without batch-normalization layers for embedded systems
Batch-normalization (BN) layers are thought to be an integrally important
layer type in today's state-of-the-art deep convolutional neural networks for
computer vision tasks such as classification and detection. However, BN layers
introduce complexity and computational overheads that are highly undesirable
for training and/or inference on low-power custom hardware implementations of
real-time embedded vision systems such as UAVs, robots and Internet of Things
(IoT) devices. They are also problematic when batch sizes need to be very small
during training, and innovations such as residual connections introduced more
recently than BN layers could potentially have lessened their impact. In this
paper we aim to quantify the benefits BN layers offer in image classification
networks, in comparison with alternative choices. In particular, we study
networks that use shifted-ReLU layers instead of BN layers. We found, following
experiments with wide residual networks applied to the ImageNet, CIFAR 10 and
CIFAR 100 image classification datasets, that BN layers do not consistently
offer a significant advantage. We found that the accuracy margin offered by BN
layers depends on the data set, the network size, and the bit-depth of weights.
We conclude that in situations where BN layers are undesirable due to speed,
memory or complexity costs, that using shifted-ReLU layers instead should be
considered; we found they can offer advantages in all these areas, and often do
not impose a significant accuracy cost.Comment: 8 pages, published IEEE conference pape