Field programmable gate array based sigmoid function implementation using differential lookup table and second order nonlinear function

Artificial neural network (ANN) is an established artificial intelligence technique that is widely used for solving numerous problems such as classification and clustering in various fields. However, the major problem with ANN is a factor of time. ANN takes a longer time to execute a huge number of neurons. In order to overcome this, ANN is implemented into hardware namely field-programmable-gate-array (FPGA). However, implementing the ANN into a field-programmable gate array (FPGA) has led to a new problem related to the sigmoid function implementation. Often used as the activation function for ANN, a sigmoid function cannot be directly implemented in FPGA. Owing to its accuracy, the lookup table (LUT) has always been used to implement the sigmoid function in FPGA. In this case, obtaining the high accuracy of LUT is expensive particularly in terms of its memory requirements in FPGA. Second-order nonlinear function (SONF) is an appealing replacement for LUT due to its small memory requirement. Although there is a trade-off between accuracy and memory size. Taking the advantage of the aforementioned approaches, this thesis proposed a combination of SONF and a modified LUT namely differential lookup table (dLUT). The deviation values between SONF and sigmoid function are used to create the dLUT. SONF is used as the first step to approximate the sigmoid function. Then it is followed by adding or deducting with the value that has been stored in the dLUT as a second step as demonstrated via simulation. This combination has successfully reduced the deviation value. The reduction value is significant as compared to previous implementations such as SONF, and LUT itself. Further simulation has been carried out to evaluate the accuracy of the ANN in detecting the object in an indoor environment by using the proposed method as a sigmoid function. The result has proven that the proposed method has produced the output almost as accurately as software implementation in detecting the target in indoor positioning problems. Therefore, the proposed method can be applied in any field that demands higher processing and high accuracy in sigmoid function outpu

Neural Network in Hardware

This dissertation describes the implementation of several neural networks built on a field programmable gate array (FPGA) and used to recognize a handwritten digit dataset – the Modified National Institute of Standards and Technology (MNIST) database. A novel hardwarefriendly activation function called the dynamic ReLU (D-ReLU) function is proposed. This activation function can decrease chip area and power of neural networks when compared to traditional activation functions at no cost to prediction accuracy. The implementations of three neural networks on FPGA are presented: 2-layer online training fully-connected neural network, 3-layer offline training fully-connected neural network, and two solutions of Super-Skinny Convolutional Neural Network (SS-CNN). The 2-layer online training fully-connected neural network was built on an FPGA with varying data width. Reducing the data width from 8 to 4 bits only reduces prediction accuracy by 11%, but the FPGA area decreases by 41%. The 3-layer offline training fully-connected neural network was built on an FPGA with both the sigmoid and the proposed D-ReLU activation functions. Compared to networks that use the sigmoid function, the proposed D-ReLU function uses 24-41% less area with no loss to prediction accuracy. Further reducing the data width of the 3-layer networks from 8 to 4 bits, the prediction accuracy only decreased by 3-5%, with area being reduced by 9-28%. The proposed sequential and parallel SS-CNN networks perform state-of-the-art (99%) recognition accuracy but with fewer layers and less neurons than prior works, for example, the LeNet-5 network. Using parameters with 8 bits of precision, the FPGA solutions of this SS-CNN show no recognition accuracy loss when compared to the 32-bit floating point software solution. In addition to high recognition accuracy, both of the proposed FPGA solutions are low power and can fit in a low cost Cyclone IVE FPGA. Moreover, these FPGA solutions have maximally 145× faster execution time than software solutions, even despite running at 97× to 120× lower clock rate. Thus, FPGA implementations of neural networks offer a high-performance, low-power alternative to traditional software methods, and the proposed novel D-ReLU activation function offers additional improvements to performance and power savings. Furthermore, the hardware solutions of the proposed SS-CNN provide a high-performance, hardware-friendly, and power efficient solution compared to other bulky convolutional neural networks

Handwritten Digit Recognition System on an FPGA

This paper describes our implementation of a multilayer perceptron (MLP) learning network on a Cyclone IVE field programmable gate array (FPGA). The MLP uses MNIST data, the Modified National Institute of Standards and Technology database of handwritten digits, to train and test the design. Working with 8-bit precision, the FPGA design has similar accuracy and execution time as the 32-bit software solution but with 144 times slower clock frequency. With power consumption being proportional to frequency, the hardware solution provides power savings at no cost in accuracy or performance. Further reducing the precision from 8 to 4 bits only reduces accuracy from 89% to 78%, with area decreasing by 41%. Thus, the FPGA implementation of the MLP learning network offers a high-performance, low power alternative to traditional software methods