Gallus, Michal

Nannarelli, Alberto

Training of deep neural networks is often constrained by the available memory and computational power. This often causes it to run for weeks even when the underlying platform is employed with multiple GPUs. In order to speed up the training and reduce space complexity the paper presents an approach of using reduced precision (8-bit) ﬂoating points for training hand-written characters classiﬁer LeNeT-5 which allows for achieving 97.10% (Top-1 and Top-5) accuracy while reducing the overall space complexity by 75% in comparison to a model using single precision ﬂoating points

Online Research Database In Technology

                       General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.  • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal   If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.     Downloaded from orbit.dtu.dk on: Apr 10, 2018Handwritten Digit Classication using 8-bit Floating Point based Convolutional NeuralNetworksGallus, Michal; Nannarelli, AlbertoPublication date:2018Document VersionPublisher's PDF, also known as Version of recordLink back to DTU OrbitCitation (APA):Gallus, M., & Nannarelli, A. (2018). Handwritten Digit Classication using 8-bit Floating Point based ConvolutionalNeural Networks. DTU Compute.  (DTU Compute Technical Report-2018, Vol. 01).Handwritten Digit Classification using 8-bit FloatingPoint based Convolutional Neural NetworksMichal Gallus and Alberto Nannarelli (supervisor)Danmarks Tekniske UniversitetLyngby, Denmarks172679@student.dtu.dkAbstract—Training of deep neural networks is often con-strained by the available memory and computational power.This often causes it to run for weeks even when the underlyingplatform is employed with multiple GPUs. In order to speed upthe training and reduce space complexity the paper presents anapproach of using reduced precision (8-bit) floating points fortraining hand-written characters classifier LeNeT-5 which allowsfor achieving 97.10% (Top-1 and Top-5) accuracy while reducingthe overall space complexity by 75% in comparison to a modelusing single precision floating points.Keywords—approximate computing, deep learningI. INTRODUCTIONImage classification has recently been dominated by Con-volutional Neural Networks (CNNs) that provide state of theart accuracy, sometimes reaching human level capabilities. Inorder to achieve better results by means of accuracy, the deepneural network models rapidly grow in size, resulting in longtraining times, high demand for memory and computationalpower. One of the most commonly seen solutions to this prob-lem concerns employing large CPU Clusters [1], or GraphicalProcessing Units (GPUs) [2] which recent models of startedincluding deep learning specific hardware units, such as TensorCores in Nvidia Volta [3]. Application Specific IntegratedCircuits (ASICs) as well as FPGA solutions have also beenproposed [4].What the majority of aforementioned solutions fail toexploit is the inherent resilience to errors associated with deeplearning applications. Because the neural networks are exposedto work with real-life noisy data while maintaining successfulresults [5], they do not require full precision range providedby the hardware in a form of 64-, 32- and even 16-bit floatingpoint numbers. Limiting the precision of underlying primitiveswhich build up the network can result in power, time andmemory savings.Nearly all of the state of the art convolutional neural net-work models spend most of the processing time on convolutionand max pooling layer [6], [7], [8]. Timespan of performingall convolution operations on the whole network amounts to80% of total time spent on training [9]. At the same time,nearly 20% of remaining time is spent on subsampling, alsoknown as max pooling operations. These on the other handare memory bound, and their speed is limited by the CPU’scache size and bandwidth of the bus.This work was done as part of the coursework in ”Design of ArithmeticProcessors” under the supervision of A. Nannarelli.In order to address this problem, this paper proposes usageof 8-bit floating point instead of single precision floating pointwhich allows to save 75% space for all trainable parameters,and possibly reduce power and computation time. To test theproposed solution the computations are ran on LeNeT-5 [10]network using MNIST hand-written images dataset [11].The paper is organised in the following manner: SectionII presents the current advancements in the field. Section IIIconcerns the main topic of the paper where the structure of 8-and 12-bit float will be presented along with the architectureknown as LeNeT-5. In section IV reader can find the resultsof training and validation of the aforementioned network usingfloats of different bit-widths. Finally the section V concludesthe paper and proposes further improvements.II. RELATED WORKDuring the design of special purpose hardware intendedfor deep neural networks processing, one of the key factors toconsider is the used precision and format of numbers used forstoring the results and performing the computations. Most ofthe literature so far has focused on using limited precision dur-ing the forward pass (also known as inference), while trainingthe network on single precision. Some of the recent works havepresented usage of custom hardware implementations [12],while others have employed the usage of FPGAs [4] for thesake of reconfigurability or using SSE3 vector instructions toperform multiple (up to 16) 8-bit operations in parallel [13].Other studies concern usage of different number format.In [14] 24-bit floating point processing units are used, while[15] uses 16-bit fixed point with stochastic rounding, achievingerrors of 0.70% on modified LeNeT-5 architecture. This resulthas later been reproduced along with introduction of dynamicfixed point by [16]. Hammerstrom [17] shows successful train-ing of the network using 16- and 8-bit fixed-point arithmetic.This work presents that both training and inference is possi-ble using not only fixed- but also low precision (down to 8-bits)floating point arithmetic. This includes direct application to notonly simple perceptron networks, but also models of highercomplexity, such as LeNeT-5 architecture which can be usedas fully functional module for Optical Character Recognition(OCR) program.III. TRAINING LOW PRECISION DEEP NEURAL NETWORKTypical implementations of deep neural networks usinggradient descent training method use 32-bit floating pointFig. 1. Representation of low precision floating points usedrepresentation of real numbers. In order to prove that suchhigh precision is not necessary for successful training andinference, the complete from-scratch C++ implementation ofLeNeT-5 has been created that would easily allow for switch-ing between underlying data type that is used for storingimage data, weights, and all intermediaries. All the operationsperformed on the data use single-precision floating pointarithmetic. Therefore the type representing floating point ofvariable precision had to have its conversion operators from-and to-32-bit float defined. These, along with other details onimplementation have been presented in the following section.A. Low-precision floating pointFloating point format is an industry standard of repre-senting real numbers. A single number consists of a sign,exponent and finally a mantissa, all portrayed in the Figure1. Exponent is used to allow for representing a wide range ofthe real numbers, while mantissa serves for assuring adequateprecision. A real number computed from its floating pointformat can be computed based on the following formula:value = (−1)sign × (1 + mantissa2M)× 2exponent−biaswhere M is the amount of bits used to represent mantissa,while bias is equal to 2E−1 − 1, E being the number of bitsused for exponent.Bit-width Exponent bits Mantisa bits32 8 2316 5 1012 5 68 4 3TABLE I. PRECISION USED DURING TRAINING AND INFERENCETable I shows all four representations that have beensubject to neural network training and testing during theexperiment.B. ImplementationSince the current CPUs do not support floats of bit-width lower than 32-bits, the need for implementing the low-precision primitives has arised. This could potentially includebuilding a class representing a minifloat and including allthe arithmetic operations on the software level. This however,would incur a very high overhead, which along with typicallylong neural network training times would result in unaccept-able amount of days and hours spent on debugging and testing.Therefore, the decision has been made to develop a primitivethat would only store the value of low-precision float, anddefine conversions from and to a single precision float. Theconversions are needed, so that all the arithmetic operationscan be performed on primitives that can benefit from hardwareacceleration.The following subsections cover how the conversions arebeing handled. Please note, that from now on, the notion ofminifloat will be used to denote a non-standard low precisionfloating point number such as 8- or 12-bit float.C. Float → minifloat conversionIn ordert to convert 32-bit floating point number to mini-float, the following steps need to be performed:1) Determine and set the sign bit2) Compute rounding of mantissaa) Determine guard, last, round and sticky bitsb) Compute G(L+R+T ) and save result to r3) Shift float mantissa right by (23− S) and save to m4) Save the minifloat mantissa mm as a sum of obtainedshift m and the rounding r5) Determine if overflow has occured using equationo = r ·m6) Extract the float exponent ef and subtract the floatbias er = ef − (27 − 1) to obtain the raw exponenter7) Compute minifloat exponent em by adding the mini-float bias and an mantissa overflow flag em = er +(2E−1 − 1) + o8) If em < 0:a) Determine the value of the guard bit (themost significant bit that will be truncated)and store it in Gub) Compute the new mantissa using equation(mm+2S)·2em+Gu where 2S is the hiddenbit, 2em shifts the value right by an absolutevalue of minifloat exponent, and addition ofGu approximates roundingc) Set minifloat exponent em to zeroElse if em > 2E − 1:a) Based on the sign, set the number to positiveor negative infinity, by setting exponent to(2E − 1) and mantissa to 0Where S and E are the bit lengths of minifloat significant andexponent respectively.D. Minifloat → float conversionConversion between minifloat and single precision float canbe summed up in the following algorithm:1) Check if the minifloat represents an infinity, and if itdoes, return the float version of it2) If the exponent em = 0 (subnormal):a) If the mantissa mm = 0 return 0b) Else, determine the position of leading onein mantissa, and save it to pc) Set float mantissa mf to mm shifted left by(23− p) bits using equation mm · 223−pd) Calculate raw exponent using equation er =−p− (2E−1 − 1)3) Else:a) Shift minifloat mantissa left by (23 − S),using formula mf = mm · 223−Sb) Calculate raw exponent by subtracting thebias using equation er = em − (2E−1 − 1)4) Compute float exponent by adding bias: ef = er +2E−1 − 15) Determine and save the sign.E. MNIST datasetThe dataset used for the training has been chosen to bethe collection of handwritten digits, collected and publishedby Lecun in [11]. Sample of examined data has been shownin Figure 2. The set consists of 60,000 training images and10,000 validation images. Each image has a size of 28 × 28and represents a digit from 0 to 9. Pixel values have beennormalized to lay in [0, 1] range. The data did not undergoany augmentation.Fig. 2. Visualisation of small subset of MNIST - the training datasetF. LeNeT-5 ArchitectureIn order to test the influence of the floating point precisionon the training and inference of recognition of handwrittencharacters, the architecture proposed by Lecun in [10] calledLeNeT-5 is used.The network consists of the following layers:1) Input Layer2) Convolution 5× 5, 6 feature maps3) MaxPooling 2× 24) Convolution 5× 5, 16 feature maps5) MaxPooling 2× 26) Convolution 5× 5, 120 feature maps7) Fully Connected, 10 feature mapsThe amount of trainable parameters, amounts to 60500.From the most common activation functions such as ReLU,sigmoid, the hyperbolic tangent has been chosen to activatefeature maps for both convolutional and fully connected layers.The network doesn’t use any special learning improvementssuch as weight decay which would slightly decrease weightsmagnitude after each epoch or learning rate decay which woulddo similar, but with respect to the learning rate parameter.Instead it maintains a constant learning rate of 0.0015. Theloss function used is a Mean-Square-Error (MSE).IV. RESULTSThe results of running simulation on different precisionspresented in Table II show that there is a very low (2.06%)loss of accuracy between usage of 8- and 32-bit floating points.On the other hand, 12-bit floating points show very littleperformance degradation (by 0.63%). At the same time the16-bit format preserves the same accuracy as single precisionformat, and as displayed in Figure 3 it has a comparable to32-bit speed of convergence.8-bit float 12-bit float 16-bit float 32-bit float97.11% 98.63% 99.18% 99.17%TABLE II. HIGHEST ACCURACY ACHIEVED DURING THE TRAININGIf the application in question can sacrifice the 2% ofaccuracy, it’ll allow for overall memory reduction by a factor of4. This implication associates with itself several consequences.First of all, when used in CPU with embedded on-chip cachememory, it allows for storing 4x as many values as in caseof single precision float. This is important especially forsubsampling layers which are memory bound. Moreover, sincein some cases CNNs are used in real-time systems such asi. e. Autonomous driving cars, usage of low-precision floatswould loosen the bandwidth requirements, or allow for sendingimages at greater resolution than in case of 32-bits, or sendingimage frames at higher rate.Furthermore, an arithmetic unit operating on the 8-bitfloating point can be proposed. Such unit would cover loweramount of area compared to ordinary 32-bit floating pointALU. Lower amount of bits decreases carry propagationtime, as well as number of partial products computed duringmultiplication, so often used when computing convolutions.Due to reduced area, more of such units could be embeddedon the chips, resulting in greater parallelism.Alternatively, since the smaller-area units use lower amountof energy, these could be used in mobile device CPUs, andcontinue the trend of off-loading simple recognition tasks fromcloud-computing to embedded systems.Fig. 3. Plot of accuracy vs. epochs elapsed during the network trainingV. SUMMARY AN FUTURE WORKSThis paper has shown that training and inference of deepneural networks can be conducted using very low-precisionfloating point primitives. This can be exploited to significantlyoptimize memory usage (up to 4 times) and to design eithera power- or compute-efficient hardware dedicated for deeplearning purposes.Nevertheless, the increase of 2% in error classificationwhen 8-bit floats are used can be unacceptable for accuracy-critical applications. Moreover, the hardware used for roundingimplementation is crucial for retaining utmost of number’sidentity, yet expensive, therefore some future works couldconcentrate on implementation of simpler rounding schemesthan those presented in the paper.Potentially, results of this work could be extended to exam-ine and cover usage of low-precision floating-point numbers indifferent than image classification deep learning applications.REFERENCES[1] J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior,P. Tucker, K. Yang, Q. V. Le et al., “Large scale distributed deepnetworks,” in Advances in neural information processing systems, 2012,pp. 1223–1231.[2] A. Coates, P. Baumstarck, Q. Le, and A. Y. Ng, “Scalable learning forobject detection with gpu hardware,” in Intelligent Robots and Systems,2009. IROS 2009. IEEE/RSJ International Conference on. IEEE, 2009,pp. 4287–4293.[3] I. Volta, “The worlds most advanced data center gpu,” URLhttps://devblogs. nvidia. com/parallelforall/inside-volta, 2017.[4] C. Farabet, B. Martini, B. Corda, P. Akselrod, E. Culurciello, andY. LeCun, “Neuflow: A runtime reconfigurable dataflow processorfor vision,” in Computer Vision and Pattern Recognition Workshops(CVPRW), 2011 IEEE Computer Society Conference on. IEEE, 2011,pp. 109–116.[5] D. Rolnick, A. Veit, S. Belongie, and N. Shavit, “Deep learning is robustto massive label noise,” arXiv preprint arXiv:1705.10694, 2017[6] K. Simonyan and A. Zisserman, “Very deep convolutional networks forlarge-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.[7] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classificationwith deep convolutional neural networks,” in Advances in neuralinformation processing systems, 2012, pp. 1097–1105.[8] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan,V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,”in Proceedings of the IEEE conference on computer vision and patternrecognition, 2015, pp. 1–9.[9] J. Cong and B. Xiao, “Minimizing computation in convolutional neuralnetworks,” in International conference on artificial neural networks.Springer, 2014, pp. 281–290.[10] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-basedlearning applied to document recognition,” Proceedings of the IEEE,vol. 86, no. 11, pp. 2278–2324, 1998.[11] Y. LeCun, “The mnist database of handwritten digits,” http://yann.lecun. com/exdb/mnist/, 1998.[12] J. Kim, K. Hwang, and W. Sung, “X1000 real-time phoneme recognitionvlsi using feed-forward deep neural networks,” in Acoustics, Speech andSignal Processing (ICASSP), 2014 IEEE International Conference on.IEEE, 2014, pp. 7510–7514.[13] V. Vanhoucke, A. Senior, and M. Z. Mao, “Improving the speed ofneural networks on cpus,” in Proc. Deep Learning and UnsupervisedFeature Learning NIPS Workshop, vol. 1, 2011, p. 4.[14] A. Iwata, Y. Yoshida, S. Matsuda, Y. Sato, and N. Suzumura, “Anartificial neural network accelerator using general purpose 24 bitsfloating point digital signal processors,” in IJCNN, vol. 2, 1989, pp.171–182.[15] S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan, “Deeplearning with limited numerical precision,” in Proceedings of the 32ndInternational Conference on Machine Learning (ICML-15), 2015, pp.1737–1746.[16] M. Courbariaux, Y. Bengio, and J.-P. David, “Training deep neu-ral networks with low precision multiplications,” arXiv preprintarXiv:1412.7024, 2014.[17] D. Hammerstrom, “A vlsi architecture for high-performance, low-cost,on-chip learning,” in Neural Networks, 1990., 1990 IJCNN InternationalJoint Conference on. IEEE, 1990, pp. 537–544.

Handwritten Digit Classiﬁcation using 8-bit Floating Point based Convolutional Neural Networks

http://orbit.dtu.dk/ws/files/143775018/tr18_01_Nannarelli_A.pdf

Handwritten Digit Classiﬁcation using 8-bit Floating Point based Convolutional Neural Networks

Abstract

Similar works

Full text

Available Versions

Online Research Database In Technology