This paper proposes an efficient hardware architecture for a function generator suitable for an artificial neural network (ANN). A spline-based approximation function is designed that provides a good trade-off between accuracy and silicon area, whilst also being inherently scalable and adaptable for numerous activation functions. This has been achieved by using a minimax polynomial and through optimal placement of the approximating polynomials based on the results of a genetic algorithm. The approximation error of the proposed method compares favourably to all related research in this field. Efficient hardware multiplication circuitry is used in the implementation, which reduces the area overhead and increases the throughput

B. Widrow

D. Zhang

D.J. Myers

G. Cauwenberghs

H. Amin

H. Faiedh

I. Koren

J. Holt

J. Pineiro

K. Basterretxea

M. Tommiska

S. Draghici

S. Haykins

S. Vassiliadis

U. Ruckert

English

Crossref

An Efficient Hardware Architecture for a Neural Network Activation Function Generator

Larkin, Daniel

Kinane, Andrew

Muresan, Valentin

O'Connor, Noel E.

Name not available

An Efficient Hardware Architecture for a NeuralNetwork Activation Function GeneratorDaniel Larkin⋆, Andrew Kinane, Valentin Muresan, and Noel O’ConnorCentre for Digital Video Processing, Dublin City University, Dublin 9, Ireland.{larkind,kinanea,muresanv,oconnorn}@eeng.dcu.ieAbstract. This paper proposes an efficient hardware architecture foran elementary function generator that is suitable for use as the activa-tion function in an artificial neural network (ANN). A spline-based ap-proximation function is designed that provides a good trade-off betweenaccuracy and ...., whilst also being inherently scalable and adaptable toother approxmations. This has been achieved by using a minimax poly-nomial and through optimisation of the placement of the splines by usinga genetic algorithm. The approximation error of the proposed activationfunction compares favourably to all related research in this field. Multi-plication circuitry is avoided through the use of distributed arithmeticthereby reducing the area overhead and power consumption of the hard-ware implementation. The proposed architecture is up to 2,751 timesfaster than a direct implementation of a sigmoid activation function onan ARM 9 processor with a comparable clock frequency.1 IntroductionArtificial neural networks (ANN) have found widespread deployment in a broadspectrum of classification, perception, association and control applications [1].However, practical ANN implementations for high dimensional data tasks, suchas multimedia analysis, computer gaming, etc, create considerable demands formicroprocessor cycles. This is due to the fact that such ANN require extremelyhigh throughput, in addition to a large number of inputs, neurons and layers. Theassociated computational complexity is highly undesirable from real time opera-tion and low power consumption perspectives. This poses considerable problemsfor constrained computing platforms (e.g. mobile devices) which suffer from lim-itations such as low computational power, low memory capacity, short batterylife and strict miniaturisation requirements. An attractive solution to this is todesign systems whereby ANN complexity can be addressed by oﬄoading pro-cessing from the host processor to dedicated hardware for general purpose ANNacceleration.There has been considerable research in both analog and digital hardwareANN implementations – [2][3][4]. Low complexity architectures have been favoured,⋆ The support of the Enterprise Ireland Informatics Initiative is gratefully acknowl-edged.2particularly for implementing the activation function. Low complexity approacheshave the benefit of allowing reduced silicon area, but this typically comes at theexpense of output performance. It is generally accepted that, as the resolutionand precision of the inputs, weights and the activation function are reduced, sotoo is the ability of the ANN to act as a universal approximator [5][6]. How-ever, in an era where large microprocessors now use half a billion transistors,the overall benefit of area savings in the order of tens to hundreds of transistorsis questionable, particularly if the output performance and scalability is com-promised. This observation has motivated us to design a high precision, powerefficient hardware activation function generator, which is capable of accommo-dating multiple activation functions and their derivatives.The rest of this paper is organised as follows: section 2 details related prior re-search in the area. Section 3 proposes the use of a minimax spline approximationscheme. Section 4 outlines an associated hardware architecture for this scheme.Section 5 details hardware synthesis results and power consumption estimates,whilst section 6 draws conclusions about the work presented.2 Related ResearchGiven its desirable non linear characteristics and ease of differentiability, a Sig-moid based activation function, such as that defined in eqn. 1, is commonlyused in neural networks [7]. However, a direct hardware implementation is notpractical as it requires excessive logic, resulting in significant power loss.y(x) =11 + e−x(1)Consequently, a number of approximations amenable to hardware implementa-tion have been developed. Since a direct look up table (LUT) implementationuses excessive memory, approaches typically fall into the following broad cat-egories: piecewise linear approximations [8][9][10][11][12][13], piecewise secondorder approximations [11] and combinatorial input/output mappings [14]. Fur-thermore, there is considerable variance within each category. For example, anA-Law companding technique is used in [8], a sum of steps approximation is usedin [9], a multiplier-less piecewise approximation is presented in [10], a recursivepiece multiplier-less approximation is presented in [13]. An elementary functiongenerator capable of multiple activation functions using a first and second orderpolynomial approximation is detailed in [11]. Recently, a combinatorial approachhas been suggested that considerably reduces the approximation error [14]. Ourapproach, which will be describe in detail in section 4, uses a minimax first orderpolynomial approximation. The use of a minimax polynomial has been suggestedbefore in the context of a floating point activation function approximation [12].However, we further minimise the maximum error and implement an area andpower efficient architecture. Our approach produces a small approximation error,whilst being suitable for implementation of multiple activation functions.32.1 Data representation and precision requirementsIn function approximation there are two sources of error, the approximationmethod error and the data representation error resulting from the use of a finitenumber of bits. To minimise area and power consumption, the minimum num-ber of bits should be chosen that results in an acceptable error. Hardware ANNreduced precision issues were previously explored in [5][6]. It was found that 10precision bits were sufficient for multi-layer perceptrons (MLP) trained via backpropagation [6]. Using fewer precision bits than this will effect the convergencespeed for on-chip learning, and in some cases may completely prevent conver-gence. The number of integer bits should be chosen based on the range of theactivation function. The functions we propose implementing (see section 5) havea maximum usable input integer range of ±8 and a maximum usable outputrange of ±1. For these reasons, we propose using 14 input bits (4 integer bitsand 10 fractional bits) and 12 output bits (2 integer bits and 10 fractional bits).This scheme was also used by [11]. Data representation and precision used inthis and other related research is shown in table 1. [DAN: VASSILIADIS SEEMTO HAVE DIFFERENT INPUT/OUTPUR RANGES AND DIFFERENT IN-TEGER/PRECISION BITS?]Input OutputTotal Range Integer Fractional Total Range Integer FractionalBits Bits Bits Bits Bits BitsMyers et al [8] 16 [-8,8[ 4 12 8 [0,1[ 1 7Alippi et al [9] not discussedAmin et al [10] 8 [-8,8[ 4 4 8 [0,1[ 1 7Vassiliadis et al [11] 14 [-4,4[ 3 10 14 [0,1[ 4 10Faiedh et al [12] Single precision floating pointBasterretxea et al [13] not discussedTommiska-337 [14] 6 [-8,8[ 3 3 7 [0,1[ 0 7Tommiska-336 [14] 6 [-8,8[ 3 3 6 [0,1[ 0 6Tommiska-236 [14] 5 [-4,4[ 2 3 6 [0,1[ 0 6Tommiska-235 [14] 5 [-4,4[ 2 3 5 [0,1[ 0 5Table 1. Data widths and precision used in related research3 Proposed Approximation SchemePolynomial approximating functions, such as a Taylor series (see eqn. 2 for aTaylor series expansion of a sigmoid function), can be used to represent anyarbitrary continuous function. To reduce the order of the approximating polyno-mial, the input domain of the function can be sub-divided into smaller intervals.This allows a polynomial of much lower order to be used to approximate eachof the of sub-intervals. The resulting composite function is known as a piecewisepolynomial or spline. Using a spline-based activation function approximationoffers the benefit that multiple activation functions can be accommodated by4merely changing the coefficients of the approximating polynomials. This impliesthat minimal extra hardware is required to support these additional functions.SigmoidTaylor =12+14∗ x−148∗ x3 +1480∗ x5 −1780640∗ x7 +311451520∗ x9... (2)A Taylor series expansion itself is unsuitable for spline approximation as theapproximation error is maximised at the boundaries of the spline i.e. large errorsare present at the points where the splines are joined. [DAN: DOESN’T REALLYEXPLAIN WHY A TAYLOR IS UNSUITABLE AS OPPOSE TO ANOTHERTECHNIQUE – AREN’T THEY ALL PIECE-WISE? ALSO MAY CONFUSETHE READER INTRODUCING TAYLOR FOR EQN 2 AND THEN NOTUSING IT, ALTHOUGH I SEE WHY YOU DID THIS.] Consequently, otherapproximating polynomials, such as least squares have been employed [11]. Wepropose using the Remez reduction algorithm to solve the minimax spline ap-proximation [15] since this approach is novel in the context of fixed point ANNactivation function approximation. A minimax polynomial minimises the maxi-mum error over a range for a given order. As will be detailed in section 5, thisgreatly improves the approximating error relative to other polynomial approx-imating schemes such as least squares. We advocate using a first order min-imax approximation, as this is capable of achieving an acceptable error (seesection 2 DAN: YOU DON’T REALLY TALK ABOUT ACCEPTABLE ER-ROR IN SECTION 2 EXCEPT IN VERY HIGH LEVEL TERMS. I SUGGESTREPHRASING OR DELETING THIS REFERENCE TO SECTION 2) witha small number of approximating polynomials. This has the further benefit ofavoiding higher order xn operations. To further reduce the number of splinesrequired to achieve a specified accuracy, we employ the common approach ofrange reduction that exploits inherent redundancies (e.g. symmetry, periodicbehaviour, etc) so that fewer splines can be used to fully represent the function.3.1 Optimisation of spline locationsThe placement of the approximating polynomials on the input range clearly hasa major bearing on the overall approximation error. The simplest approach isto evenly distribute the polynomials over the approximating range. However,astute placement can reduce the approximating error, although the potentialsearch space for optimal placement is large. For example, when using 5 polyno-mials in a 0 to 8 range with a precision of 10−3, there are in the order of 217possible combinations for the location of the polynomials. To address this largesearch space issue, we propose using a genetic algorithm (GA) to find the op-timum location of the approximating polynomials. Unlike an exhaustive search,this solution is scalable even if the input range were to become extremely large(for example, double precision floating point). DAN: OK BUT WHY A GASPECIFICALLY AS OPPOSE OT ANOTHER OPT APPROACH?The GA was implemented using Matlab [16]. The fitness function firstlyuses the Remez reduction algorithm [DAN: NEED A ONE LINER EXPLANA-TION OF WHAT’S HAPPENING HERE] to calculate the minimax polynomial5coefficients for each candidate in the population. Using these coefficients, theminimax spline approximation to the chosen activation function (e.g. Sigmoid)is constructed. The mean and maximum errors are then calculated from thisapproximation, using at least 106 samples. The GA explores the search spacewhilst attempting to minimise these mean and maximum error values.As is usually the case, the GA needed extensive tuning through trial and errorexploration of the different input parameters. Initial population sizes of 10 to1,000 were considered, along with extensive investigation into different crossoverfunctions and different mutation functions. Our best results were obtaind usingarithmetic crossover and a multi-point non uniform mutation with 5 mutationpoints.4 Hardware ArchitectureSTILL WORKING ON THIS SECTION4.1 Range Reduction & Reconstruction4.2 Calculation of Approximating PolynomialsArray MultiplierDistributed Arithmetic Multiplication5 Results - still needs work5.1 Approximation Error[DAN: NEED TO EXPLAIN HERE WHAT FIGURES IN THE TABLE MEANANDWHY IT IS A FAIR COMPARISON. E.G. WHY IS FAIR TO COMPAREOUR APPROACH TO THAT OF [11] – DIFFERENT RANGES AND NOTCLEAR HOW MANY SEGMENTS THEY USE] We used the proposed sig-moid activation function approximation to compare our approach to previousresearch. We tested our method using 2 to 8 approximating polynomial seg-ments. The maximum and average error results for these tests can be seen intable 2. When using 5 or more segments, our method outperforms all relatedresearch. Comparing our 4 segment implementation with the first and secondorder 4 segment approach of [11], our implementation gives an error that is al-most 4 times smaller. The improvement obtained by using the genetic algorithmis apparent from comparing our five segment approach with the five segmentapproach of [12]. Our approach is 36% more accurate despite the fact that [12]employs a single precision floating point data representation.[DAN CHANGE ”OUR APPROACH” IN TABLE TO ”PROPOSED AP-PROACH”]6Design Range Maximum AverageError ErrorMyers et al [8] [-8,8[ 0.0490 0.0247Alippi et al [9] [-8,8[ 0.0189 0.0087Amin et al [10] [-8,8[ 0.0189 0.0059Vassiliadis et al (First Order) [11] [-4,4[ 0.0180 0.0035Vassiliadis et al (Second Order) [11] [-4,4[ 0.0180 0.0026Faiedh et al [12] [-5,5] 0.0050 n/aBasterretxea et al (q=3) [13] [-8,8[ 0.0222 0.0077Tommiska (337) [14] [-8,8[ 0.0039 0.0017Tommiska (336) [14] [-8,8[ 0.0077 0.0033Tommiska (236) [14] [-4,4[ 0.0077 0.0040Tommiska (235) [14] [-4,4[ 0.0151 0.0069Our approach (2 segments) [-8,8[ 0.0158 0.0068Our approach (3 segments) [-8,8[ 0.0078 0.0038Our approach (4 segments) [-8,8[ 0.0047 0.0024Our approach (5 segments) [-8,8[ 0.0032 0.0017Our approach (6 segments) [-8,8[ 0.0023 0.0012Our approach (7 segments) [-8,8[ 0.0017 0.0009Our approach (8 segments) [-8,8[ 0.0013 0.0009Table 2. Maximum and average Sigmoid Approximation errors5.2 Hardware Implementation ResultsThe design was captured in Verilog HDL and synthesised using Synopsys De-sign Compiler and Synplicity Synplify Pro for a 90nm TSMC ASIC library andXilinx Virtex 2 FPGA respectively. Power consumption estimates for the ASIClibrary were generated using Synopsys Prime Power. A summary of the synthesisresults for a five segment implementation can be seen in table 3. The distributedarithmetic approach uses 15% less area than the array multiplier for an ASICimplementation.[DAN: AGAIN DESCRIBE WHY THIS IS A VALID COMPARISON – RE-MEMBER REVIEWERS ARE UNLIKELY TO BE HARDWARE BODS]Area Max Frequency Average Power[Gates] [MHz] [mWatts]ASIC - Array multiplier 1917 250 0.1423mW -this figure is a problem!!ASIC - Distributed Arithmetic 1635 250 0.152mWFPGA - Array multiplier 2800 40 n/aFPGA - Distributed Arithmetic ? 40 n/aTable 3. Synthesis ResultsDISCUSS FPGA IMPLEMENTATION7COMPARE TO RELATED RESEARCH - ATTEMPT TO NORMALISETHEIR RESULTSBoth array multiplier and distributed arithmetic architectures complete pro-cessing within one clock cycle. This compares very favourable to a direct imple-mentation of a sigmoid activation function on an ARM 9 processor, which weprofiled (see table 4) as requiring a minimum of 395 clock cycles [DAN: PRO-VIDE MORE DETAIL - OPTIMISED IMPLEMENTATION, OR THROWNTOGETHER?]. Profiling also revealed that even a first order spline approxi-mation requires a minimum of 55 clock cycles on an ARM 9 processor. [DAN:WHY IS THIS INTERESTING .. COS WE’RE SO MUCH BETTER THANTHAT AND YET ONLY NEED 1 CYCLE ... HAND HOLD THE READERTHROUGH THIS EVEN AT THE RISK OF SOUNDING ”OBVIOUS”]Design Floating Point Instruction ClockCo-processor count cyclesDirect implementation not present 1,688 2,550Direct implementation present 114 3951st order spline approximation not present 323 4481st order spline approximation present 55 163Table 4. Sigmoid activation function profiling on an ARM 920 Processor6 Future work and conclusionsThis paper presented a high precision, scalable hardware architecture for an ac-tivation function generator. It represents preliminary work toward a completepower efficient hardware ANN accelerator for mobile platform suitable for han-dling high dimensional data sets such as those used in multimedia applications.In the future, we plan to extend the number of activation functions supported,along with modifications to the genetic algorithm to compensate for any biasesintroduced from the rounding of the coefficients to a finite word length [DAN:NEED TO EXPLAIN OR REMOVE]. In addition, we also intend to conduct athorough investigation on the suitability of using floating point representation.References[1] Benard Widrow, David E. Rumelhart, and Michael A. Lehr, “Neural networks:Applications in industry, business and science,” Communications of the ACM,vol. 37, no. 3, pp. 93–105, Mar. 1994.[2] G. Cauwenberghs and M. Bayoumi, Learning on Silicon - Adaptive VLSI NeuralSystems, Kluwer Academic, 1999.[3] David Zhang and Sankar K. Pal, Neural Networks and Systolic Array Design,World Scientific, New Jersey, 2002.8[4] U. Ruckert, “ULSI architectures for artificial neural networks,” IEEE Micro, vol.22, no. 3, pp. 10–19, May 2002.[5] Sorin Draghici, “On the capabilities of neural networks using limited precisionweights,” Neural Networks, vol. 15, no. 3, pp. 395 – 414, Apr. 2002.[6] J.L. Holt and J.N Hwang, “Finite precision error analysis of neural networkhardwareimplementations,” vol. 42, no. 3, pp. 280 – 291, Mar. 1993.[7] Simon Haykins, Neural Networks - A Comprehensive Foundation, Prentice-Hall,1999.[8] D.J. Myers and R.A Hutchinson, “Efficient implementation of piecewise linearactivation function for digital vlsi neural networks,” in Electronics Letters, Nov.1989, vol. 25, pp. 1662–1663.[9] C. Alippi and G. Storti-Gajani, “Simple approximation of sigmoid functions:Realistic design of digital vlsi neural networks,” in Proceedings of the IEEE Int’lSymp. Circuits and Systems, June 1991, pp. 1505–1508.[10] H. Amin, K.M. Curtis, and B.R Hayes Gill, “Piecewise linear approximationapplied to nonlinear function of a neural network,” IEE Proceedings Circuits,Devices and Systems, vol. 144, pp. 313 – 317, Dec. 1997.[11] S. Vassiliadis, Ming Zhang, and J.G. Delgado-Frias, “Elementary function gen-erators for neural-network emulators,” IEEE Transactions on Neural Networks,vol. 11, no. 6, pp. 1438 – 1449, Nov. 2000.[12] H. Faiedh, Z. Gafsi, and K. Besbes, “Digital hardware implementation of sigmoidfunction and its derivative for artificial neural networks,” in Proceedings. of the13th International Conference on Microelectronics, Rabat, Morocco, Oct. 2001,pp. 189 – 192.[13] K. Basterretxea, J.M. Tarela, and I. del Campo, “Approximation of sigmoidfunction and the derivative for hardware implementation of artificial neurons,”IEE Proceedings of Circuits, Devices and Systems, vol. 151, no. 1, pp. 18–24, Feb.2004.[14] M.T. Tommiska, “Efficient digital implementation of the sigmoid function forreprogrammable logic,” IEE Proceedings Computers and Digital Techniques, vol.150, pp. 403 – 411, Nov. 2003.[15] J.A. Pineiro, S.F. Oberman, J.M. Muller, and J.D. Bruguera, “High-Speed Func-tion Approximation Using a Minimax Quadratic Interpolator,” vol. 54, pp. 304 –318, Mar. 2005.[16] “The MathWorks - MATLAB and Simulink for Technical Computing,”http://www.mathworks.com/.

An efficient hardware architecture for a neural network activation function generator

DCU Online Research Access Service

Approximation of sigmoid function and the derivative for hardware implementation of artiﬁcial neurons,”

Digital hardware implementation of sigmoid function and its derivative for artiﬁcial neural networks,”

Eﬃcient digital implementation of the sigmoid function for reprogrammable logic,”

Eﬃcient implementation of piecewise linear activation function for digital vlsi neural networks,”

Elementary function generators for neural-network emulators,”

Finite precision error analysis of neural network hardwareimplementations,”

High-Speed Function Approximation Using a

Learning on Silicon - Adaptive VLSI Neural Systems,

Neural Networks - A Comprehensive Foundation,

Neural networks: Applications in industry, business and science,”

On the capabilities of neural networks using limited precision weights,”

Piecewise linear approximation applied to nonlinear function of a neural network,”

Simple approximation of sigmoid functions: Realistic design of digital vlsi neural networks,”

ULSI architectures for artiﬁcial neural networks,”

O\u27Connor, Noel E.

Irish Universities

An efficient hardware architecture for a neural network activation function generator

Abstract

Similar works

Full text

Available Versions

Crossref

Name not available

DCU Online Research Access Service

Irish Universities