An efficient implementation of lattice-ladder multilayer perceptrons in field programmable gate arrays

The implementation efficiency of electronic systems is a combination of conflicting requirements, as increasing volumes of computations, accelerating the exchange of data, at the same time increasing energy consumption forcing the researchers not only to optimize the algorithm, but also to quickly implement in a specialized hardware. Therefore in this work, the problem of efficient and straightforward implementation of operating in a real-time electronic intelligent systems on field-programmable gate array (FPGA) is tackled. The object of research is specialized FPGA intellectual property (IP) cores that operate in a real-time. In the thesis the following main aspects of the research object are investigated: implementation criteria and techniques. The aim of the thesis is to optimize the FPGA implementation process of selected class dynamic artificial neural networks. In order to solve stated problem and reach the goal following main tasks of the thesis are formulated: rationalize the selection of a class of Lattice-Ladder Multi-Layer Perceptron (LLMLP) and its electronic intelligent system test-bed – a speaker dependent Lithuanian speech recognizer, to be created and investigated; develop dedicated technique for implementation of LLMLP class on FPGA that is based on specialized efficiency criteria for a circuitry synthesis; develop and experimentally affirm the efficiency of optimized FPGA IP cores used in Lithuanian speech recognizer. The dissertation contains: introduction, four chapters and general conclusions. The first chapter reveals the fundamental knowledge on computer-aideddesign, artificial neural networks and speech recognition implementation on FPGA. In the second chapter the efficiency criteria and technique of LLMLP IP cores implementation are proposed in order to make multi-objective optimization of throughput, LLMLP complexity and resource utilization. The data flow graphs are applied for optimization of LLMLP computations. The optimized neuron processing element is proposed. The IP cores for features extraction and comparison are developed for Lithuanian speech recognizer and analyzed in third chapter. The fourth chapter is devoted for experimental verification of developed numerous LLMLP IP cores. The experiments of isolated word recognition accuracy and speed for different speakers, signal to noise ratios, features extraction and accelerated comparison methods were performed. The main results of the thesis were published in 12 scientific publications: eight of them were printed in peer-reviewed scientific journals, four of them in a Thomson Reuters Web of Science database, four articles – in conference proceedings. The results were presented in 17 scientific conferences

FPGA implementation of a Restricted Boltzmann Machine for handwriting recognition

Despite the recent success of neural network in the research eld, the num- ber of resulting applications for non-academic settings is very limited. One setback for its popularity is that neural networks are typically implemented as software running on a general-purpose processor. The time complexity of the software implementation is usually O(n2). As a result, neural net- works are inadequate to meet the scalability and performance requirements for commercial or industrial uses. Several research works have dealt with accelerating neural networks on Field-Programmable Gate Arrays (FPGAs), particularly for Restricted Boltzmann Machines (RBMs) | a very popular and hardware-friendly neural network model. However, when using their implementations for handwriting recognition, there are two major setbacks. First, the implementations assume that the sizes of the neural networks are symmetric, while the size of RBM model for handwriting recognition is in fact highly asymmetric. Second, these implementations cannot t a model with a visible layer larger than 512 nodes on a single FPGA. Thus, they are highly ine cient when apply to handwriting recognition application. In this thesis, a new framework was proposed for an RBM with asymmetric weights optimizing for handwriting recognition. The framework is tested on an Altera Stratix IV GX(EP4SGX230KF40C2) FPGA running at 100 MHz. The resources support a complete RBM model of 784 by 10 nodes. The experimental results show the computational speed of 4 billion connection- update-per-second and a speed-up of 134 fold with I/O time and a speed- up of 161 fold without I/O time compared with an optimized MATLAB implementation running on a 2.50 GHz Intel processor. Compared with previous works, our implementation is able to achieve a much higher speed- up while maintaining comparable resources used

Dynamical Systems in Spiking Neuromorphic Hardware

Dynamical systems are universal computers. They can perceive stimuli, remember, learn from feedback, plan sequences of actions, and coordinate complex behavioural responses. The Neural Engineering Framework (NEF) provides a general recipe to formulate models of such systems as coupled sets of nonlinear differential equations and compile them onto recurrently connected spiking neural networks – akin to a programming language for spiking models of computation. The Nengo software ecosystem supports the NEF and compiles such models onto neuromorphic hardware. In this thesis, we analyze the theory driving the success of the NEF, and expose several core principles underpinning its correctness, scalability, completeness, robustness, and extensibility. We also derive novel theoretical extensions to the framework that enable it to far more effectively leverage a wide variety of dynamics in digital hardware, and to exploit the device-level physics in analog hardware. At the same time, we propose a novel set of spiking algorithms that recruit an optimal nonlinear encoding of time, which we call the Delay Network (DN). Backpropagation across stacked layers of DNs dramatically outperforms stacked Long Short-Term Memory (LSTM) networks—a state-of-the-art deep recurrent architecture—in accuracy and training time, on a continuous-time memory task, and a chaotic time-series prediction benchmark. The basic component of this network is shown to function on state-of-the-art spiking neuromorphic hardware including Braindrop and Loihi. This implementation approaches the energy-efficiency of the human brain in the former case, and the precision of conventional computation in the latter case

Compensação digital de distorções da fibra em sistemas de comunicação óticos de longa distância

The continuous increase of traffic demand in long-haul communications motivated the network operators to look for receiver side techniques to mitigate the nonlinear effects, resulting from signal-signal and signal-noise interaction, thus pushing the current Capacity boundaries. Machine learning techniques are a very hot-topic with given proofs in the most diverse applications. This dissertation aims to study nonlinear impairments in long-haul coherent optical links and the current state of the art in DSP techniques for impairment mitigation as well as the integration of machine learning strategies in optical networks. Starting with a simplified fiber model only impaired by ASE noise, we studied how to integrate an ANN-based symbol estimator into the signal pipeline, enabling to validate the implementation by matching the theoretical performance. We then moved to nonlinear proof of concept with the incorporation of NLPN in the fiber link. Finally, we evaluated the performance of the estimator under realistic simulations of Single and Multi- Channel links in both SSFM and NZDSF fibers. The obtained results indicate that even though it may be hard to find the best architecture, Nonlinear Symbol Estimator networks have the potential to surpass more conventional DSP strategies.O aumento contínuo de tráfego nas comunicações de longo-alcance motivou os operadores de rede a procurar técnicas do lado do receptor para atenuar os efeitos não lineares resultantes da interacção sinal-sinal e sinal-ruído, alargando assim os limites da capacidade do sistema. As técnicas de aprendizagem-máquina são um tópico em ascenção com provas dadas nas mais diversas aplicações e setores. Esta dissertação visa estudar as principais deficiências nas ligações de longo curso e o actual estado da arte em técnicas de DSP para mitigação das mesmas, bem como a integração de estratégias de aprendizagem-máquina em redes ópticas. Começando com um modelo simplificado de fibra apenas perturbado pelo ruído ASE, estudámos como integrar um estimador de símbolos baseado em ANN na cadeia do prodessamento de sinal, conseguindo igualar o desempenho teórico. Procedemos com uma prova de conceito perante não linearidades com a incorporação do ruído de fase não linear na propagação. Finalmente, avaliamos o desempenho do estimador com simulações realistas de links Single e Multi canal tanto em fibras SSFM como NZDSF. Os resultados obtidos indicam que apesar da dificuldade de encontrar a melhor arquitectura, a estimação não linear baseada em redes neuronais têm o potencial para ultrapassar estratégias DSP mais convencionais.Mestrado em Engenharia Eletrónica e Telecomunicaçõe