6 research outputs found
Using the IBM Analog In-Memory Hardware Acceleration Kit for Neural Network Training and Inference
Analog In-Memory Computing (AIMC) is a promising approach to reduce the
latency and energy consumption of Deep Neural Network (DNN) inference and
training. However, the noisy and non-linear device characteristics, and the
non-ideal peripheral circuitry in AIMC chips, require adapting DNNs to be
deployed on such hardware to achieve equivalent accuracy to digital computing.
In this tutorial, we provide a deep dive into how such adaptations can be
achieved and evaluated using the recently released IBM Analog Hardware
Acceleration Kit (AIHWKit), freely available at https://github.com/IBM/aihwkit.
The AIHWKit is a Python library that simulates inference and training of DNNs
using AIMC. We present an in-depth description of the AIHWKit design,
functionality, and best practices to properly perform inference and training.
We also present an overview of the Analog AI Cloud Composer, that provides the
benefits of using the AIHWKit simulation platform in a fully managed cloud
setting. Finally, we show examples on how users can expand and customize
AIHWKit for their own needs. This tutorial is accompanied by comprehensive
Jupyter Notebook code examples that can be run using AIHWKit, which can be
downloaded from https://github.com/IBM/aihwkit/tree/master/notebooks/tutorial
The Effect of Batch Normalization on Noise Resistant Property of Deep Learning Models
The fast execution speed and energy efficiency of analog hardware have made them a strong contender for deploying deep learning models at the edge. However, there are concerns about the presence of analog noise which causes changes to the models’ weight, leading to performance degradation of deep learning models, despite their inherent noise-resistant characteristics. The effect of the popular batch normalization layer (BatchNorm) on the noise-resistant ability of deep learning models is investigated in this work. This systematic study has been carried out by first training different models with and without the BatchNorm layer on the CIFAR10 and the CIFAR100 datasets. The weights of the resulting models are then injected with analog noise, and the performance of the models on the test dataset is obtained and compared. The results show that the presence of the BatchNorm layer negatively impacts the noise-resistant property of deep learning models, i.e., ResNet44 and VGG16 models with BatchNorm layers trained with the CIFAR10 dataset have an average normalized inference accuracy of 41.32% and 10.75% respectively compared to 91.95% and 93.80% obtained for same ResNet44 and VGG16 model without the BatchNorm layer respectively. Furthermore, the impact of the BatchNorm layer also grows with the increase of the number of BatchNorm layers
Impact of L1 Batch Normalization on Analog Noise Resistant Property of Deep Learning Models
Analog hardware has become a popular choice for machine learning on
resource-constrained devices recently due to its fast execution and energy
efficiency. However, the inherent presence of noise in analog hardware and the
negative impact of the noise on deployed deep neural network (DNN) models limit
their usage. The degradation in performance due to the noise calls for the
novel design of DNN models that have excellent noiseresistant property,
leveraging the properties of the fundamental building block of DNN models. In
this work, the use of L1 or TopK BatchNorm type, a fundamental DNN model
building block, in designing DNN models with excellent noise-resistant property
is proposed. Specifically, a systematic study has been carried out by training
DNN models with L1/TopK BatchNorm type, and the performance is compared with
DNN models with L2 BatchNorm types. The resulting model noise-resistant
property is tested by injecting additive noise to the model weights and
evaluating the new model inference accuracy due to the noise. The results show
that L1 and TopK BatchNorm type has excellent noise-resistant property, and
there is no sacrifice in performance due to the change in the BatchNorm type
from L2 to L1/TopK BatchNorm type
Using the IBM analog in-memory hardware acceleration kit for neural network training and inference
Analog In-Memory Computing (AIMC) is a promising approach to reduce the latency and energy consumption of Deep Neural Network (DNN) inference and training. However, the noisy and non-linear device characteristics and the non-ideal peripheral circuitry in AIMC chips require adapting DNNs to be deployed on such hardware to achieve equivalent accuracy to digital computing. In this Tutorial, we provide a deep dive into how such adaptations can be achieved and evaluated using the recently released IBM Analog Hardware Acceleration Kit (AIHWKit), freely available at https://github.com/IBM/aihwkit. AIHWKit is a Python library that simulates inference and training of DNNs using AIMC. We present an in-depth description of the AIHWKit design, functionality, and best practices to properly perform inference and training. We also present an overview of the Analog AI Cloud Composer, a platform that provides the benefits of using the AIHWKit simulation in a fully managed cloud setting along with physical AIMC hardware access, freely available at https://aihw-composer.draco.res.ibm.com. Finally, we show examples of how users can expand and customize AIHWKit for their own needs. This Tutorial is accompanied by comprehensive Jupyter Notebook code examples that can be run using AIHWKit, which can be downloaded from https://github.com/IBM/aihwkit/tree/master/notebooks/tutorial