The deep learning community focuses on training networks for a better accuracy on GPU servers. However, bringing this technology to consumer products requires inference adaptation of suchInstruction networks for low-energy, small-memory, and computationally constrained edge devices. ARM CPU is one of the important components of edge devices, but a clear comparison between the existinginference frameworks is missing. We provide minimal preliminaries about ARM CPU architecture and briefly mention the difference between the existing inference frameworks to evaluate them based on performance versus usability trade-offs

Courville, Vanessa

Nia, Vahid Partovi

English

Waterloo Library Journal Publishing Service (University of Waterloo, Canada)

Deep Learning Inference frameworks for ARM CPUVanessa Courville Huawei Noah’s Ark Lab, CanadaVahid Partovi Nia Huawei Noah’s Ark Lab, CanadaAbstractThe deep learning community focuses on training networks for abetter accuracy on GPU servers. However, bringing this technol-ogy to consumer products requires inference adaptation of suchnetworks for low-energy, small-memory, and computationally con-strained edge devices. ARM CPU is one of the important compo-nents of edge devices, but a clear comparison between the existinginference frameworks is missing. We provide minimal preliminar-ies about ARM CPU architecture and briefly mention the differencebetween the existing inference frameworks to evaluate them basedon performance versus usability trade-offs.1 IntroductionReal-time performance of deep model inference models is crucialfor most end-user applications. As neural network architecturescontinue to become deeper and more complex, how these modelsare deployed on hardware devices has a direct impact on responsetime. As a result, optimized and architecture-dependent implemen-tations are typically required to ensure maximum performance andusability.Many open-source inference frameworks exists that target dif-ferent types of hardware devices. These devices include server-grade CPUs, such as Intel x86 architectures, smaller embeddedCPUs such as ARM, domain-specific ASICs, or even reprogrammableFPGAs. In all cases, the devices chosen are directly dependent onthe application.One of the most common and readily-available hardware archi-tectures used for neural network inference is the ARM CPU whichare based on Reduced Instruction Set Computer, ARM architec-tures are designed to support a reduced set of instructions, result-ing in a smaller die size, reduced energy requirements and allowingit to run at higher frequencies. As a result, ARM processors are typ-ically the go-to architecture for most small consumer electronics.2 ARM ArchitectureThe ARM CPU architecture is typically made up of a multi-coredesign wherein each core contains a 32-bit/64-bit CPU, a NEONSIMD accelerator, a floating point unit, and a series of cache, asshown in Figure 1NEON SIMDEngineFloatingPointCPU32-bit /64-bitI-Cache D-CacheFig. 1: Example of typical ARM architecture.2.1 NEON SIMD EngineSingle instruction, multiple data (SIMD) programming is a popu-lar form of parallel processing supported by most modern CPUarchitectures. Based on the concept of vectorization, a parallelprogramming paradigm where an algorithm is modified to run asingle instruction on multiple different sets of data points concur-rently. SIMD processors are typically represented as specializedhardware accelerator components of a CPU. For ARM architec-tures, the SIMD accelerators is known as NEON [1]. The NEONaccelerator can be programmed using either C-intrinsics or assem-bly programming. Figure 2 shows the flow of a simple SIMD pro-cessor.Compute UnitDataInstructionResultCompute UnitCompute UnitCompute UnitResult Result ResultData Data DataFig. 2: Example of typical SIMD architecture.The NEON SIMD accelerator is a key component of ARM ar-chitectures and is vital for any performance-driven applications. Itallows for an other-wise small, inefficient processor to maximizecompute capability and significantly increase the number of con-current operations possible.The computations required in most neural network operationscan be treated as an embarrassingly parallel workload as a verylarge number of independent operations take place. This makes itvery straight-forward to vectorize most neural networks operations,making it a perfect application for SIMD acceleration.2.2 big.LITTLEARM systems with big.LITTLE architectures [2] are heterogeneoussystems made up of multi-core processors of different compute ca-pabilities. Typically, the system will contain both smaller, power-saving processors (LITTLE) as well as larger, more compute-intensiveprocessors (big). Typically used for consumer electronics, thesearchitectures are designed to maximize both power savings andperformance by allowing the designer to pick and choose whichprocessor to use depending on the application requirements. Anexample of a typical big.LITTLE architecture is shown in Figure 3CortexA76CortexA55CortexA55L2 CacheCache Coherent InterconnectCortexA55CortexA55L2 CacheCortexA76IO CoherentMasterbig Cores LITTLE CoresFig. 3: Example of an ARM big.LITTLE architecture.These types of architectures are typically found in most em-bedded devices where performance and power-efficiency are im-perative. Many mobile phones currently available in the market usethe ARM big.LITTLE architecture as a means of splitting up taskson the device. Performance-critical applications will run on the ’big’processors whereas background applications will run on the ’LIT-TLE’ processors.As many neural network applications run on mobile devices,it is vital that any inference framework designed to target mobileplatforms must be able to efficiently take advantage of this archi-tecture.3 QuantizationAs neural networks continue to evolve, the number of applicationsfor typical consumer electronics continues to increase. Applica-tions such as speech recognition, object character recognition andmodel personalization all require large, complex neural networksto provide high accuracy. These models require significant datastorage and compute capability which are constrained on edge de-vices.In order to run large neural network models on these edgedevices, the models must be compressed to not only reduce theoverall storage size, but also reduce the number of computationsneeded to ensure that results can be achieved within a reasonabledelay and save energy consumption.The most common technique of model compression is quanti-zation. Quantization involves reducing precision, of either weights,activations, or both in a typical neural network [3]. Say a full pre-cision model contains weights and activations that are each 32-bitfloating point values. A quantized version of this model could in-volve replacing the 32-bit float weights/activations with quantized8-bit integers. This not only reduced storage by a factor of 4 times,but also simplifies the compute required as there are no longer anyfloating-point multiply-accumulate operations taking place. Insteadthese can be replaced by simple 8-bit integer multiplications, effec-tively simplifying the hardware, reducing the compute latency, andsaving energy.In order to successfully support these compressed models, in-ference frameworks targeting edge devices must be able to supportthese quantization techniques. Therefore, an inference frameworkmust support different quantized data precisions for both weightsand activations.4 Existing Inference FrameworksThere is currently no shortage of available open-source inferenceframeworks targeting ARM architectures. These frameworks varyin terms of performance, flexibility, usability, and data-precision sup-port.4.1 ncnnTencent’s neural network inference framework (ncnn) [4] is highlyoptimized to target mobile platforms using ARM CPUs. The frame-work targets the NEON accelerator using a combination of C intrin-sics and low-level assembly instructions to maximize performance.Written entirely in C++, it contains a big.LITTLE multi-core sched-uler, and does not depend on any third party library. The frameworkcan also support 8-bit quantization, including 8-bit compute opera-tions and half precision storage. It can also be extended to supportGPU offload. However, currently it does not support very low-bitprecision such as binary networks.4.2 mnnAlibaba’s Mobile Neural Network (mnn) [5] inference frameworkstrength is in its versatility. It can be integrated into multiple high-level frameworks such as TensorFlow, ONNX, and Caffe. Muchlike ncnn, it targets performance using hand-coded NEON assem-bly code, and supports heterogeneous environments by offloadingcertain operations to the GPU. MNN does not support 8-bit quan-tization, but does support full half-precision compute (using ARMv8.2 new features) differentiating it from ncnn.4.3 NeoCPUAmazon Web Services’ NeoCPU [6]was designed to be genericand does not target a specific hardware architecture. Instead,the framework supports multiple hardware devices including In-tel, AMD and ARM CPUs because the framework is built upon aTVM deep learning compiler stack. The code does not use any in-trinsic of assembly code for performance optimization, but insteadfocuses on algorithmic improvements such as transformation tech-niques, layout optimizations and search space optimization. Al-though versatile, it does not currently support low-bit precision net-works.4.4 BMXNet-v2Hasso Platner Institute’s BMXNet-v2 [7] is an extension of the pop-ular MXNet framework targeting binary and quantized networks.This framework differs from the other frameworks by the fact thatit supports a wide array of quantization schemes including, 2 to31 bits. Although it does not target performance, its wide range ofsupport is intended to allow researchers to evaluate different quan-tization schemes.4.5 dabnnJD AI Research Group’s dabnn [8] is a binary neural network in-ference framework targeting mobile platforms. The framework isdesigned for highly optimized binary convolutions using ARM as-sembly code. It integrates easily into ONNX by providing a con-version tool. Although optimized for binary networks, DABNN doesnot support 8-bit or half precision data formats.4.6 ARM Compute LibraryARM’s neural network compute library [9] is a set of pre-built binaryfunctions for common neural network functions. Provided by ARMdirectly, the library is highly-optimized and takes full advantage ofthe NEON SIMD architecture. It supports some low-precision op-erations such as 8-bit quantized convolutions but currently doesnot support binary or half-precision networks. It is packaged as astand-alone library and does not have a pre-defined integration intohigh-level frameworks.4.7 Inference Framework ComparisonEach of the inference frameworks evaluated have varying sets oftrade-offs. Table 1 summarizes the features of all networks evalu-ated.Feature ncnn mnn Neo BMXNet dabnn CLNEON X X x X X Xb.L. X x x x x xGPU X X x x x Xhalf Storage X x x x x8-bit X x x X x Xbinary x x x X X xAndroid X X x X X XTF/ONNX x X X x X XOpt x x X x x xTable 1: Inference framework feature support comparison5 DiscussionMany companies and academics have attempted to design performance-optimized inference frameworks. These frameworks vary as sometarget versatility, with the intent of being able to use the sameframework for different hardware devices, while others target per-formance by hand-tuning the implementation using assembly code,or by focusing on supporting low-bit quantized networks.However, what is lacking is a comprehensive framework whosesole focus is performance targeting resource constrained edge de-vices. In order to get optimal performance on these devices, aframework must be able to take full advantage of the hardwareresources available to it, including being able to maximize perfor-mance on the SIMD accelerator, be able to schedule jobs acrossa heterogeneous architecture including a multi-core big.LITTLE ar-chitecture or GPU offload. It must have full support or model com-pression techniques like quantization with multiple different sup-ported precision types. In addition to these essentials, the frame-work should also take advantage of algorithmic optimization tech-niques as seen by the NeoCPU framework.Industry and academia alike would significantly benefit fromsuch a framework as it would ensure that all neural networks mod-els can be tested using realistic consumer-driven hardware andconditions.6 AcknowledgementWe appreciate fruitful technical discussions with Huawei Cloud CoreShanghai colleagues Gang Chi and Pengcheng Tang. We alsothank Yan Zhou and Yanhui Geng for their support throughout theproject.References[1] ARM. Introducing neon development article. Technical report,2009.[2] ARM. big.little technology: The future of mobile. Technicalreport, 2013.[3] Raghuraman Krishnamoorthi. Quantizing deep convolutionalnetworks for efficient inference: A whitepaper. arXiv preprintarXiv:1806.08342, 2018.[4] Tencent. ncnn. https://github.com/Tencent/ncnn,2019.[5] Alibaba. mnn. https://github.com/alibaba/MNN, 2019.[6] Yu Li Shaarma Wang Liu, Wang. Optimizing cnn model in-ference on cpus. USENIX Annual Technical Conference 20191025-1040, 2019.[7] Hasso Platner Institute. Bmxnet-v2. https://github.com/hpi-xnor/BMXNet-v2, 2019.[8] JD AI Research Group. dabnn. https://github.com/JDAI-CV/dabnn, 2019.[9] ARM-Software. Arm compute library. https://github.com/ARM-software/ComputeLibrary, 2019.

Deep Learning Inference Frameworks for ARM CPU

https://openjournals.uwaterloo.ca/index.php/vsl/article/download/1645/2014

Deep Learning Inference Frameworks for ARM CPU

Abstract

Similar works

Full text

Available Versions

Waterloo Library Journal Publishing Service (University of Waterloo, Canada)