Design and training for combinational neural-logic systems

Centre for Multimedia Signal Processing, Department of Electronic and Information Engineering2006-2007 > Academic research: refereed > Publication in refereed journalVersion of RecordPublishe

Design Space Exploration of Neural Network Activation Function Circuits

The widespread application of artificial neural networks has prompted researchers to experiment with FPGA and customized ASIC designs to speed up their computation. These implementation efforts have generally focused on weight multiplication and signal summation operations, and less on activation functions used in these applications. Yet, efficient hardware implementations of nonlinear activation functions like Exponential Linear Units (ELU), Scaled Exponential Linear Units (SELU), and Hyperbolic Tangent (tanh), are central to designing effective neural network accelerators, since these functions require lots of resources. In this paper, we explore efficient hardware implementations of activation functions using purely combinational circuits, with a focus on two widely used nonlinear activation functions, i.e., SELU and tanh. Our experiments demonstrate that neural networks are generally insensitive to the precision of the activation function. The results also prove that the proposed combinational circuit-based approach is very efficient in terms of speed and area, with negligible accuracy loss on the MNIST, CIFAR-10 and IMAGENET benchmarks. Synopsys Design Compiler synthesis results show that circuit designs for tanh and SELU can save between 3.13-7.69 and 4.45-8:45 area compared to the LUT/memory-based implementations, and can operate at 5.14GHz and 4.52GHz using the 28nm SVT library, respectively. The implementation is available at: https://github.com/ThomasMrY/ActivationFunctionDemo.Comment: 5 pages, 5 figures, 16 conferenc

Generalized disjunction decomposition for evolvable hardware

Evolvable hardware (EHW) refers to self-reconfiguration hardware design, where the configuration is under the control of an evolutionary algorithm (EA). One of the main difficulties in using EHW to solve real-world problems is scalability, which limits the size of the circuit that may be evolved. This paper outlines a new type of decomposition strategy for EHW, the “generalized disjunction decomposition” (GDD), which allows the evolution of large circuits. The proposed method has been extensively tested, not only with multipliers and parity bit problems traditionally used in the EHW community, but also with logic circuits taken from the Microelectronics Center of North Carolina (MCNC) benchmark library and randomly generated circuits. In order to achieve statistically relevant results, each analyzed logic circuit has been evolved 100 times, and the average of these results is presented and compared with other EHW techniques. This approach is necessary because of the probabilistic nature of EA; the same logic circuit may not be solved in the same way if tested several times. The proposed method has been examined in an extrinsic EHW system using the$(1 + lambda)$evolution strategy. The results obtained demonstrate that GDD significantly improves the evolution of logic circuits in terms of the number of generations, reduces computational time as it is able to reduce the required time for a single iteration of the EA, and enables the evolution of larger circuits never before evolved. In addition to the proposed method, a short overview of EHW systems together with the most recent applications in electrical circuit design is provided

On evolution of relatively large combinational logic circuits

Evolvable hardware (EHW) (Yao and Higuchi, 1999) is a technique introduced to automatically design circuits where the circuit configuration is carried out by evolutionary algorithms. One of the main difficulties in using EHW to solve real-world problems is the scalability. Until now, several strategies have been proposed to avoid this problem, but none of them completely tackle the issue. In this paper three different methods for evolving the most complex circuits have been tested for their scalability. These methods are bi-directional incremental evolution (SO-BIE); generalised disjunction decomposition (GD-BIE) and evolutionary strategies (ES) with dynamic mutation rate. In order to achieve the generalised conclusions the chosen approaches were tested using multipliers, traditionally used in EHW, but also logic circuits taken from MCNC (Yang, 1991) benchmark library and randomly generated circuits. The analysis of the approaches demonstrated that PLA-based ES is capable of evolving logic circuits of up to 12 inputs. The use of SO-BIE allows the generation of fully functional circuits of 14 inputs and GD-BIE is estimated to be able to evolve circuits of 21 inputs

Efficient Neural Network Implementations on Parallel Embedded Platforms Applied to Real-Time Torque-Vectoring Optimization Using Predictions for Multi-Motor Electric Vehicles

The combination of machine learning and heterogeneous embedded platforms enables new potential for developing sophisticated control concepts which are applicable to the field of vehicle dynamics and ADAS. This interdisciplinary work provides enabler solutions -ultimately implementing fast predictions using neural networks (NNs) on field programmable gate arrays (FPGAs) and graphical processing units (GPUs)- while applying them to a challenging application: Torque Vectoring on a multi-electric-motor vehicle for enhanced vehicle dynamics. The foundation motivating this work is provided by discussing multiple domains of the technological context as well as the constraints related to the automotive field, which contrast with the attractiveness of exploiting the capabilities of new embedded platforms to apply advanced control algorithms for complex control problems. In this particular case we target enhanced vehicle dynamics on a multi-motor electric vehicle benefiting from the greater degrees of freedom and controllability offered by such powertrains. Considering the constraints of the application and the implications of the selected multivariable optimization challenge, we propose a NN to provide batch predictions for real-time optimization. This leads to the major contribution of this work: efficient NN implementations on two intrinsically parallel embedded platforms, a GPU and a FPGA, following an analysis of theoretical and practical implications of their different operating paradigms, in order to efficiently harness their computing potential while gaining insight into their peculiarities. The achieved results exceed the expectations and additionally provide a representative illustration of the strengths and weaknesses of each kind of platform. Consequently, having shown the applicability of the proposed solutions, this work contributes valuable enablers also for further developments following similar fundamental principles.Some of the results presented in this work are related to activities within the 3Ccar project, which has received funding from ECSEL Joint Undertaking under grant agreement No. 662192. This Joint Undertaking received support from the European Union’s Horizon 2020 research and innovation programme and Germany, Austria, Czech Republic, Romania, Belgium, United Kingdom, France, Netherlands, Latvia, Finland, Spain, Italy, Lithuania. This work was also partly supported by the project ENABLES3, which received funding from ECSEL Joint Undertaking under grant agreement No. 692455-2

Combinational neural logic system and its industrial application on hypoglycemia monitoring system

In this paper, a combinational neural logic network (NLN) with the neural-Logic-AND, -OR and -NOT gates is applied on the development of non-invasive hypoglycemia monitoring system. It is an alarm system which measured physiological parameters of electrocardiogram (ECG) signal and determine the onset of hypoglycemia by use of proposed NLN. Due to different nature of application, conventional neural networks (NNs) with common structure may not always guarantee the optimal solution. Based on knowledge of application, the proposed NLN is designed systematically in order to incorporate the characteristics of application into the structure of proposed network. The parameter of the proposed NLN will be trained by hybrid particle swarm optimization with wavelet mutation (HPSOWM). The proposed NLN will be practically analyzed using real data sets collected from 15 children (569 data sets) with Type 1 diabetes at the Department of Health, Government of Western Australia. By using the proposed method, the detection performance is enhanced. Compared with other conventional NNs, the proposed NLN gives better performance in terms of sensitivity and specificity. © 2013 IEEE