Stochastic rounding and reduced-precision fixed-point arithmetic for solving neural ordinary differential equations

Although double-precision floating-point arithmetic currently dominates high-performance computing, there is increasing interest in smaller and simpler arithmetic types. The main reasons are potential improvements in energy efficiency and memory footprint and bandwidth. However, simply switching to lower-precision types typically results in increased numerical errors. We investigate approaches to improving the accuracy of reduced-precision fixed-point arithmetic types, using examples in an important domain for numerical computation in neuroscience: the solution of Ordinary Differential Equations (ODEs). The Izhikevich neuron model is used to demonstrate that rounding has an important role in producing accurate spike timings from explicit ODE solution algorithms. In particular, fixed-point arithmetic with stochastic rounding consistently results in smaller errors compared to single precision floating-point and fixed-point arithmetic with round-to-nearest across a range of neuron behaviours and ODE solvers. A computationally much cheaper alternative is also investigated, inspired by the concept of dither that is a widely understood mechanism for providing resolution below the least significant bit (LSB) in digital signal processing. These results will have implications for the solution of ODEs in other subject areas, and should also be directly relevant to the huge range of practical problems that are represented by Partial Differential Equations (PDEs).Comment: Submitted to Philosophical Transactions of the Royal Society

A Survey of Spiking Neural Network Accelerator on FPGA

Due to the ability to implement customized topology, FPGA is increasingly used to deploy SNNs in both embedded and high-performance applications. In this paper, we survey state-of-the-art SNN implementations and their applications on FPGA. We collect the recent widely-used spiking neuron models, network structures, and signal encoding formats, followed by the enumeration of related hardware design schemes for FPGA-based SNN implementations. Compared with the previous surveys, this manuscript enumerates the application instances that applied the above-mentioned technical schemes in recent research. Based on that, we discuss the actual acceleration potential of implementing SNN on FPGA. According to our above discussion, the upcoming trends are discussed in this paper and give a guideline for further advancement in related subjects

Digital Implementation of Bio-Inspired Spiking Neuronal Networks

Spiking Neural Network as the third generation of artificial neural networks offers a promising solution for future computing, prosthesis, robotic and image processing applications. This thesis introduces digital designs and implementations of building blocks of a Spiking Neural Networks (SNNs) including neurons, learning rule, and small networks of neurons in the form of a Central Pattern Generator (CPG) which can be used as a module in control part of a bio-inspired robot. The circuits have been developed using Verilog Hardware Description Language (VHDL) and simulated through Modelsim and compiled and synthesised by Altera Qurtus Prime software for FPGA devices. Astrocyte as one of the brain cells controls synaptic activity between neurons by providing feedback to neurons. A novel digital hardware is proposed for neuron-synapseastrocyte network based on the biological Adaptive Exponential (AdEx) neuron and Postnov astrocyte cell model. The network can be used for implementation of large scale spiking neural networks. Synthesis of the designed circuits shows that the designed astrocyte circuit is able to imitate its biological model and regulate the synapse transmission, successfully. In addition, synthesis results confirms that the proposed design uses less than 1% of available resources of a VIRTEX II FPGA which saves up to 4.4% of FPGA resources in comparison to other designs. Learning rule is an essential part of every neural network including SNN. In an SNN, a special type of learning called Spike Timing Dependent Plasticity (STDP) is used to modify the connection strength between the spiking neurons. A pair-based STDP (PSTDP) works on pairs of spikes while a Triplet-based STDP (TSTDP) works on triplets of spikes to modify the synaptic weights. A low cost, accurate, and configurable digital architectures are proposed for PSTDP and TSTDP learning models. The proposed circuits have been compared with the state of the art methods like Lookup Table (LUT), and Piecewise Linear approximation (PWL). The circuits can be employed in a large-scale SNN implementation due to their compactness and configurability. Most of the neuron models represented in the literature are introduced to model the behavior of a single neuron. Since there is a large number of neurons in the brain, a population-based model can be helpful in better understanding of the brain functionality, implementing cognitive tasks and studying the brain diseases. Gaussian Wilson-Cowan model as one of the population-based models represents neuronal activity in the neocortex region of the brain. A digital model is proposed for the GaussianWilson-Cowan and examined in terms of dynamical and timing behavior. The evaluation indicates that the proposed model is able to generate the dynamical behavior as the original model is capable of. Digital architectures are implemented on an Altera FPGA board. Experimental results show that the proposed circuits take maximum 2% of the resources of a Stratix Altera board. In addition, static timing analysis indicates that the circuits can work in a maximum frequency of 244 MHz

An FPGA platform for real-time simulation of spiking neuronal networks

In the last years, the idea to dynamically interface biological neurons with artificial ones has become more and more urgent. The reason is essentially due to the design of innovative neuroprostheses where biological cell assemblies of the brain can be substituted by artificial ones. For closed-loop experiments with biological neuronal networks interfaced with in silico modeled networks, several technological challenges need to be faced, from the low-level interfacing between the living tissue and the computational model to the implementation of the latter in a suitable form for real-time processing. Field programmable gate arrays (FPGAs) can improve flexibility when simple neuronal models are required, obtaining good accuracy, real-time performance, and the possibility to create a hybrid system without any custom hardware, just programming the hardware to achieve the required functionality. In this paper, this possibility is explored presenting a modular and efficient FPGA design of an in silico spiking neural network exploiting the Izhikevich model. The proposed system, prototypically implemented on a Xilinx Virtex 6 device, is able to simulate a fully connected network counting up to 1,440 neurons, in real-time, at a sampling rate of 10 kHz, which is reasonable for small to medium scale extra-cellular closed-loop experiments

Central nervous system: overall considerations based on hardware realization of digital spiking silicon neurons (dssns) and synaptic coupling

The Central Nervous System (CNS) is the part of the nervous system including the brain and spinal cord. The CNS is so named because the brain integrates the received information and influences the activity of different sections of the bodies. The basic elements of this important organ are: neurons, synapses, and glias. Neuronal modeling approach and hardware realization design for the nervous system of the brain is an important issue in the case of reproducing the same biological neuronal behaviors. This work applies a quadratic-based modeling called Digital Spiking Silicon Neuron (DSSN) to propose a modified version of the neuronal model which is capable of imitating the basic behaviors of the original model. The proposed neuron is modeled based on the primary hyperbolic functions, which can be realized in high correlation state with the main model (original one). Really, if the high-cost terms of the original model, and its functions were removed, a low-error and high-performance (in case of frequency and speed-up) new model will be extracted compared to the original model. For testing and validating the new model in hardware state, Xilinx Spartan-3 FPGA board has been considered and used. Hardware results show the high-degree of similarity between the original and proposed models (in terms of neuronal behaviors) and also higher frequency and low-cost condition have been achieved. The implementation results show that the overall saving is more than other papers and also the original model. Moreover, frequency of the proposed neuronal model is about 168 MHz, which is significantly higher than the original model frequency, 63 MHz

Posit and Floating-Point Based Izhikevich Neuron: A Comparison of Arithmetic

Preprint submitted to Elsevier. It has not been certified by peer review.Reduced precision number formats are gaining popularity in many areas of computational science, due to their potential to improve energy efficiency, silicon use, and speed. However, this is often at the expense of introducing arithmetic errors which affect the accuracy of a system. The optimal balance must be struck, judiciously choosing a number format using as few bits as possible, while minimising accuracy loss. In this study, we examine one such format, posit arithmetic as a replacement for floating-point when conducting spiking neuron simulations, specifically using the Izhikevich neuron model. This model is capable of simulating complex neural firing behaviours, 20 of which were originally identified by Izhikevich and are used in this study. We compare the accuracy, spike count, and spike timing of the two arithmetic systems at different bit-depths against a 64-bit floating-point gold-standard. Additionally, we test a rescaled set of Izhikevich equations to mitigate against arithmetic errors by taking advantage of posit arithmetic’s tapered accuracy.Our findings indicate that there is no difference in performance between 32-bit posit, 32-bit floating-point, and our reference standard for 95% of the tested firing types. However, at 16-bit, both arithmetic systems diverge from the 64-bit reference, albeit non-uniformly. For instance, the posit implementation demonstrates an accumulated spike timing error of 0.5ms over a 1000ms simulation compared to 9ms for floating-point – an 18x improvement using posit arithmetic for regular (tonic) spiking. This finding holds particular importance given the prevalence of this particular firing type in specific regions of the brain. Furthermore, when we rescale the neuron equations, this error is eliminated altogether. Hence, our results demonstrate that posit arithmetic is not only a viable replacement for 64-bit floating-point in these simulations, it can do so while using 4× fewer bits. As a Posit Arithmetic Unit has similar area to a Floating Point Unit with the same bit width, this constitutes a significant saving of hardware resources while maintaining full accuracy compared to 64-bit floating-point.TFH was part funded by Sundance Multiprocesssor Ltd., UK and an EPSRC Doctoral Training Partnerships (DTP) grant. JK was funded by the EPSRC (grant EP/V052241/1)

Massively-parallel bit-serial neural networks for fast epilepsy diagnosis: a feasibility study

There are about 1% of the world population suffering from the hidden disability known as epilepsy and major developing countries are not fully equipped to counter this problem. In order to reduce the inconvenience and danger of epilepsy, different methods have been researched by using a artificial neural network (ANN) classification to distinguish epileptic waveforms from normal brain waveforms. This paper outlines the aim of achieving massive ANN parallelization through a dedicated hardware using bit-serial processing. The design of this bit-serial Neural Processing Element (NPE) is presented which implements the functionality of a complete neuron using variable accuracy. The proposed design has been tested taking into consideration non-idealities of a hardware ANN. The NPE consists of a bit-serial multiplier which uses only 16 logic elements on an Altera Cyclone IV FPGA and a bit-serial ALU as well as a look-up table. Arrays of NPEs can be driven by a single controller which executes the neural processing algorithm. In conclusion, the proposed compact NPE design allows the construction of complex hardware ANNs that can be implemented in a portable equipment that suits the needs of a single epileptic patient in his or her daily activities to predict the occurrences of impending tonic conic seizures

Spiking Neural Networks: Modification and Digital Implementation

Real-time large-scale simulation of biological systems is a challenging task due to nonlinear functions describing biochemical reactions in the cells. Being fast, cost and power efficient alongside of capability to work in parallel have made hardware an attractive choice for simulation platform. This thesis proposes a neuromorphic platform for online Spike Timing Dependant Plasticity (STDP) learning, based on the COordinate Rotation DIgital Computer (CORDIC) algorithms. The implemented platform comprises two main components. First, the Izhikevich neuron model is modified for implementation using the CORDIC algorithm and simulated to ensure the model accuracy. Afterwards, the model was described as hardware and implemented on Field Programmable Gate Array (FPGA). Second, the STDP learning algorithm is adapted and optimized using the CORDIC method, synthesized for hardware, and implemented to perform on-FPGA online learning on a network of CORDIC Izhikevich neurons to demonstrate competitive Hebbian learning. The implementation results are compared with the original model and state-of-the-art to verify accuracy, effectiveness, and higher speed of the system. These comparisons confirm that the proposed neuromorphic system offers better performance and higher accuracy while being straightforward to implement and suitable to scale. New findings show that astrocytes are important parts of the information processing in brain and believed to be responsible for some brain diseases such as Alzheimer and Epilepsy. Astrocytes generate Ca$^{2+}$ waves and release neuro-transmitters over a large area. To study astrcoytes, one need to simulate large number of biologically realistic models of these cells alongside neuron models. Software simulation is flexible but slow. This thesis proposes a high-speed and low-cost digital hardware to replicate biological-plausible astrocyte and glutamate release mechanism. The nonlinear terms of these models were calculated using high-precision and cost-efficient algorithms. Subsequently, the modified models were simulated to study and validate their functions. Several hardware were developed by setting different constraints to investigate trade-offs and achieve best possible design. As proof of concept, the design was implemented on a FPGA device. Hardware implementation results confirmed the ability of the design to replicate biological cells in detail with high accuracy. As for performance, the proposed design turned out to be far more faster and area efficient than previously published works that targeted digital hardware for biological-plausible astrocytes. Spiking neurons, the models that mimic the biological cells in the brain, are described using ordinary differential equations. A common method to numerically solve these equations is Euler\u27s method. An important factor that has a significant impact on the performance and cost of the hardware implementation or software simulation of spiking neural networks and yet its importance has been neglected in the published literature, is the time step in Euler\u27s method. In this thesis, first the Izhikevich neuron\u27s accuracy as a function of the time step was measured. It was uncovered that the threshold time step that Izhikevich neuron becomes unstable is an exponential function of the input current. Software simulation performance, including total computational time and memory usage were compared for different time steps. Afterwards, the model was synthesized and implemented on the FPGA. Hardware performance metrics such as speed, area and power consumption were measured for each time step. Results indicated that time step has a negative linear effect on the performance. It was concluded that by determining maximum input current to the neuron, larger time steps comparable to those used in the previous works could be employed