Towards Robust Design and Training of Deep Neural Networks

Currently neural networks run as software, which typically requires expensive GPU resources. As the adoption of deep learning continues for a more diverse range of applications, direct hardware implemented neural networks (HNN) will provide deep learning solutions at far lower hardware requirements. However, Gaussian noise along hardware connections degrades model accuracy, an issue this research seeks to resolve using a novel analog error correcting code (ECC). To aid in developing noise tolerant deep neural networks (DNN), this research also investigates the impact of loss functions on training. This involves alternating multiple loss functions throughout training, aiming to prevent local optimals. The effects on training time and final accuracy are then analyzed. This research investigates analog ECCs and loss function variation to allow for future noise tolerant HNN networks. ECC results demonstrate three to five decibel improvements to model accuracy when correcting Gaussian noise. Loss variation results demonstrate a correlation between loss function similarity and training performance. Other correlations are also presented and addressed

Towards Robust Design and Training of Deep Neural Networks

Currently neural networks run as software, which typically requires expensive GPU resources. As the adoption of deep learning continues for a more diverse range of applications, direct hardware implemented neural networks (HNN) will provide deep learning solutions at far lower hardware requirements. However, Gaussian noise along hardware connections degrades model accuracy, an issue this research seeks to resolve using a novel analog error correcting code (ECC). To aid in developing noise tolerant deep neural networks (DNN), this research also investigates the impact of loss functions on training. This involves alternating multiple loss functions throughout training, aiming to prevent local optimals. The effects on training time and final accuracy are then analyzed. This research investigates analog ECCs and loss function variation to allow for future noise tolerant HNN networks. ECC results demonstrate three to five decibel improvements to model accuracy when correcting Gaussian noise. Loss variation results demonstrate a correlation between loss function similarity and training performance. Other correlations are also presented and addressed

Application of the Intuitionistic Fuzzy InterCriteria Analysis Method with Triples to a Neural Network Preprocessing Procedure

The approach of InterCriteria Analysis (ICA) was applied for the aim of reducing the set of variables on the input of a neural network, taking into account the fact that their large number increases the number of neurons in the network, thus making them unusable for hardware implementation. Here, for the first time, with the help of the ICA method, correlations between triples of the input parameters for training of the neural networks were obtained. In this case, we use the approach of ICA for data preprocessing, which may yield reduction of the total time for training the neural networks, hence, the time for the network’s processing of data and images

An investigation into adaptive power reduction techniques for neural hardware

In light of the growing applicability of Artificial Neural Network (ANN) in the signal processing field [1] and the present thrust of the semiconductor industry towards lowpower SOCs for mobile devices [2], the power consumption of ANN hardware has become a very important implementation issue. Adaptability is a powerful and useful feature of neural networks. All current approaches for low-power ANN hardware techniques are ‘non-adaptive’ with respect to the power consumption of the network (i.e. power-reduction is not an objective of the adaptation/learning process). In the research work presented in this thesis, investigations on possible adaptive power reduction techniques have been carried out, which attempt to exploit the adaptability of neural networks in order to reduce the power consumption. Three separate approaches for such adaptive power reduction are proposed: adaptation of size, adaptation of network weights and adaptation of calculation precision. Initial case studies exhibit promising results with significantpower reduction

Hardware Implementation of Soft Computing Approaches for an Intelligent Wall-following Vehicle

Soft computing techniques are generally well-suited for vehicular control systems that are usually modeled by highly nonlinear differential equations and working in unstructured environment. To demonstrate their applicability, two intelligent controllers based upon fuzzy logic theories and neural network paradigms are designed for performing a wall-following task and an autonomous parking task. Based on performance and flexibility considerations, the two controllers are implemented onto a reconfigurable hardware platform, namely a Field Programmable Gate Array (FPGA). As the number of comparative studies of these two embedded controllers designed for the same application is limited in the literature, one of the main goals of this research work has been to evaluate and compare the two controllers in terms of hardware resource requirements, operational speeds and trajectory tracking errors in following different pre-defined trajectories. The main advantages and disadvantages of each of the controllers are presented and discussed in details. Challenging issues for implementation of the controllers on the FPGA platform are also highlighted. As the two controllers exhibit benefits and drawbacks under different circumstances, this research suggests as well a hybrid controller scheme as an attempt to integrate the benefits of both control units. To evaluate its performance, the hybrid controller is tested on the same pre-defined trajectories and the corresponding results are compared to that of the fuzzy logic and the neural network based controllers. For further demonstration of the capabilities of the wall-following controllers in other applications, the fuzzy logic and the neural network controllers are used in a parallel parking system. We see this work to be a stepping stone for further research work aiming at real world implementation of the controllers on Application Specified Integrated Circuit (ASIC) type of environment

Autonomously Reconfigurable Artificial Neural Network on a Chip

Artificial neural network (ANN), an established bio-inspired computing paradigm, has proved very effective in a variety of real-world problems and particularly useful for various emerging biomedical applications using specialized ANN hardware. Unfortunately, these ANN-based systems are increasingly vulnerable to both transient and permanent faults due to unrelenting advances in CMOS technology scaling, which sometimes can be catastrophic. The considerable resource and energy consumption and the lack of dynamic adaptability make conventional fault-tolerant techniques unsuitable for future portable medical solutions. Inspired by the self-healing and self-recovery mechanisms of human nervous system, this research seeks to address reliability issues of ANN-based hardware by proposing an Autonomously Reconfigurable Artificial Neural Network (ARANN) architectural framework. Leveraging the homogeneous structural characteristics of neural networks, ARANN is capable of adapting its structures and operations, both algorithmically and microarchitecturally, to react to unexpected neuron failures. Specifically, we propose three key techniques --- Distributed ANN, Decoupled Virtual-to-Physical Neuron Mapping, and Dual-Layer Synchronization --- to achieve cost-effective structural adaptation and ensure accurate system recovery. Moreover, an ARANN-enabled self-optimizing workflow is presented to adaptively explore a "Pareto-optimal" neural network structure for a given application, on the fly. Implemented and demonstrated on a Virtex-5 FPGA, ARANN can cover and adapt 93% chip area (neurons) with less than 1% chip overhead and O(n) reconfiguration latency. A detailed performance analysis has been completed based on various recovery scenarios

Efficient multiprocessing architectures for spiking neural network emulation based on configurable devices

The exploration of the dynamics of bioinspired neural networks has allowed neuroscientists to understand some clues and structures of the brain. Electronic neural network implementations are useful tools for this exploration. However, appropriate architectures are necessary due to the extremely high complexity of those networks. There has been an extraordinary development in reconfigurable computing devices within a short period of time especially in their resource availability, speed, and reconfigurability (FPGAs), which makes these devices suitable to emulate those networks. Reconfigurable parallel hardware architecture is proposed in this thesis in order to emulate in real time complex and biologically realistic spiking neural networks (SNNs). Some relevant SNN models and their hardware approaches have been studied, and analyzed in order to create an architecture that supports the implementation of these SNN models efficiently. The key factors, which involve flexibility in algorithm programmability, high performance processing, low area and power consumption, have been taken into account. In order to boost the performance of the proposed architecture, several techniques have been developed: time to space mapping, neural virtualization, flexible synapse-neuron mapping, specific learning and execution modes, among others. Besides this, an interface unit has been developed in order to build a bio-inspired system, which can process sensory information from the environment. The spiking-neuron-based system combines analog and digital multi-processor implementations. Several applications have been developed as a proof-of-concept in order to show the capabilities of the proposed architecture for processing this type of information.L'estudi de la dinàmica de les xarxes neuronals bio-inspirades ha permès als neurocientífics entendre alguns processos i estructures del cervell. Les implementacions electròniques d'aquestes xarxes neuronals són eines útils per dur a terme aquest tipus d'estudi. No obstant això, l'alta complexitat de les xarxes neuronals requereix d'una arquitectura apropiada que pugui simular aquest tipus de xarxes. Emular aquest tipus de xarxes en dispositius configurables és possible a causa del seu extraordinari desenvolupament respecte a la seva disponibilitat de recursos, velocitat i capacitat de reconfiguració (FPGAs ). En aquesta tesi es proposa una arquitectura maquinari paral·lela i configurable per emular les complexes i realistes xarxes neuronals tipus spiking en temps real. S'han estudiat i analitzat alguns models de neurones tipus spiking rellevants i les seves implementacions en maquinari , amb la finalitat de crear una arquitectura que suporti la implementació d'aquests models de manera eficient . S'han tingut en compte diversos factors clau, incloent flexibilitat en la programació d'algorismes, processament d'alt rendiment, baix consum d'energia i àrea. S'han aplicat diverses tècniques en l'arquitectura desenvolupada amb el propòsit d'augmentar la seva capacitat de processament. Aquestes tècniques són: mapejat de temps a espai, virtualització de les neurones, mapeig flexible de neurones i sinapsis, modes d'execució, i aprenentatge específic, entre d'altres. A més, s'ha desenvolupat una unitat d'interfície de dades per tal de construir un sistema bio-inspirat, que pot processar informació sensorial del medi ambient. Aquest sistema basat en neurones tipus spiking combina implementacions analògiques i digitals. S'han desenvolupat diverses aplicacions usant aquest sistema com a prova de concepte, per tal de mostrar les capacitats de l'arquitectura proposada per al processament d'aquest tipus d'informació

A Practical Investigation into Achieving Bio-Plausibility in Evo-Devo Neural Microcircuits Feasible in an FPGA

Many researchers has conjectured, argued, or in some cases demonstrated, that bio-plausibility can bring about emergent properties such as adaptability, scalability, fault-tolerance, self-repair, reliability, and autonomy to bio-inspired intelligent systems. Evolutionary-developmental (evo-devo) spiking neural networks are a very bio-plausible mixture of such bio-inspired intelligent systems that have been proposed and studied by a few researchers. However, the general trend is that the complexity and thus the computational cost grow with the bio-plausibility of the system. FPGAs (Field- Programmable Gate Arrays) have been used and proved to be one of the flexible and cost efficient hardware platforms for research' and development of such evo-devo systems. However, mapping a bio-plausible evo-devo spiking neural network to an FPGA is a daunting task full of different constraints and trade-offs that makes it, if not infeasible, very challenging. This thesis explores the challenges, trade-offs, constraints, practical issues, and some possible approaches in achieving bio-plausibility in creating evolutionary developmental spiking neural microcircuits in an FPGA through a practical investigation along with a series of case studies. In this study, the system performance, cost, reliability, scalability, availability, and design and testing time and complexity are defined as measures for feasibility of a system and structural accuracy and consistency with the current knowledge in biology as measures for bio-plausibility. Investigation of the challenges starts with the hardware platform selection and then neuron, cortex, and evo-devo models and integration of these models into a whole bio-inspired intelligent system are examined one by one. For further practical investigation, a new PLAQIF Digital Neuron model, a novel Cortex model, and a new multicellular LGRN evo-devo model are designed, implemented and tested as case studies. Results and their implications for the researchers, designers of such systems, and FPGA manufacturers are discussed and concluded in form of general trends, trade-offs, suggestions, and recommendations