A Survey of Robotics Control Based on Learning-Inspired Spiking Neural Networks

Biological intelligence processes information using impulses or spikes, which makes those living creatures able to perceive and act in the real world exceptionally well and outperform state-of-the-art robots in almost every aspect of life. To make up the deficit, emerging hardware technologies and software knowledge in the fields of neuroscience, electronics, and computer science have made it possible to design biologically realistic robots controlled by spiking neural networks (SNNs), inspired by the mechanism of brains. However, a comprehensive review on controlling robots based on SNNs is still missing. In this paper, we survey the developments of the past decade in the field of spiking neural networks for control tasks, with particular focus on the fast emerging robotics-related applications. We first highlight the primary impetuses of SNN-based robotics tasks in terms of speed, energy efficiency, and computation capabilities. We then classify those SNN-based robotic applications according to different learning rules and explicate those learning rules with their corresponding robotic applications. We also briefly present some existing platforms that offer an interaction between SNNs and robotics simulations for exploration and exploitation. Finally, we conclude our survey with a forecast of future challenges and some associated potential research topics in terms of controlling robots based on SNNs

Toward Building Hybrid Biological/in silico Neural Networks for Motor Neuroprosthetic Control

WOS: 000370402900001PubMed ID: 26321943In this article, we introduce the Bioinspired Neuroprosthetic Design Environment (BNDE) as a practical platform for the development of novel brain-machine interface (BMI) controllers, which are based on spiking model neurons. We built the BNDE around a hard real-time system so that it is capable of creating simulated synapses from extra-cellularly recorded neurons to model neurons. In order to evaluate the practicality of the BNDE for neuroprosthetic control experiments, a novel, adaptive BMI controller was developed and tested using real-time closed-loop simulations. The present controller consists of two in silico medium spiny neurons, which receive simulated synaptic inputs from recorded motor cortical neurons. In the closed-loop simulations, the recordings from the cortical neurons were imitated using an external, hardware-based neural signal synthesizer. By implementing a reward-modulated spike timing-dependent plasticity rule, the controller achieved perfect target reach accuracy for a two-target reaching task in one-dimensional space. The BNDE combines the flexibility of software-based spiking neural network (SNN) simulations with powerful online data visualization tools and is a low-cost, PC-based, and all-in-one solution for developing neurally inspired BMI controllers. We believe that the BNDE is the first implementation, which is capable of creating hybrid biological/in silico neural networks for motor neuroprosthetic control and utilizes multiple CPU cores for computationally intensive real-time SNN simulations.Bogazici University BAP Grants [10XD3]; Bogazici University Life Sciences and Technologies Research Center [09K120520]This research was supported by Bogazici University BAP Grants #10XD3 and Bogazici University Life Sciences and Technologies Research Center #09K120520

A biologically plausible embodied model of action discovery

During development, animals can spontaneously discover action-outcome pairings enabling subsequent achievement of their goals. We present a biologically plausible embodied model addressing key aspects of this process. The biomimetic model core comprises the basal ganglia and its loops through cortex and thalamus. We incorporate reinforcement learning (RL) with phasic dopamine supplying a sensory prediction error, signalling “surprising” outcomes. Phasic dopamine is used in a cortico-striatal learning rule which is consistent with recent data. We also hypothesized that objects associated with surprising outcomes acquire “novelty salience” contingent on the predicability of the outcome. To test this idea we used a simple model of prediction governing the dynamics of novelty salience and phasic dopamine. The task of the virtual robotic agent mimicked an in vivo counterpart (Gancarz et al., 2011) and involved interaction with a target object which caused a light flash, or a control object which did not. Learning took place according to two schedules. In one, the phasic outcome was delivered after interaction with the target in an unpredictable way which emulated the in vivo protocol. Without novelty salience, the model was unable to account for the experimental data. In the other schedule, the phasic outcome was reliably delivered and the agent showed a rapid increase in the number of interactions with the target which then decreased over subsequent sessions. We argue this is precisely the kind of change in behavior required to repeatedly present representations of context, action and outcome, to neural networks responsible for learning action-outcome contingency. The model also showed cortico-striatal plasticity consistent with learning a new action in basal ganglia. We conclude that action learning is underpinned by a complex interplay of plasticity and stimulus salience, and that our model contains many of the elements for biological action discovery to take place

Robust learning algorithms for spiking and rate-based neural networks

Inspired by the remarkable properties of the human brain, the fields of machine learning, computational neuroscience and neuromorphic engineering have achieved significant synergistic progress in the last decade. Powerful neural network models rooted in machine learning have been proposed as models for neuroscience and for applications in neuromorphic engineering. However, the aspect of robustness is often neglected in these models. Both biological and engineered substrates show diverse imperfections that deteriorate the performance of computation models or even prohibit their implementation. This thesis describes three projects aiming at implementing robust learning with local plasticity rules in neural networks. First, we demonstrate the advantages of neuromorphic computations in a pilot study on a prototype chip. Thereby, we quantify the speed and energy consumption of the system compared to a software simulation and show how on-chip learning contributes to the robustness of learning. Second, we present an implementation of spike-based Bayesian inference on accelerated neuromorphic hardware. The model copes, via learning, with the disruptive effects of the imperfect substrate and benefits from the acceleration. Finally, we present a robust model of deep reinforcement learning using local learning rules. It shows how backpropagation combined with neuromodulation could be implemented in a biologically plausible framework. The results contribute to the pursuit of robust and powerful learning networks for biological and neuromorphic substrates

Hierarchical reinforcement learning in a biologically plausible neural architecture

Humans and other animals have an impressive ability to quickly adapt to unfamiliar environments, with only minimal feedback. Computational models have been able to provide intriguing insight into these processes, by making connections between abstract computational theories of reinforcement learning (RL) and neurophysiological data. However, the ability of these models falls well below the level of real neural systems, thus it is clear that there are important aspects of the neural computation not being captured by our models. In this work we explore how new developments from the computational study of RL can be expanded to the realm of neural modelling. Specifically, we examine the field of hierarchical reinforcement learning (HRL), which extends RL by dividing the RL process into a hierarchy of actions, where higher level decisions guide the choices made at lower levels. The advantages of HRL have been demonstrated from a computational perspective, but HRL has never been implemented in a neural model. Thus it is unclear whether HRL is a purely abstract theory, or whether it could help explain the RL ability of real brains. Here we show that all the major components of HRL can be implemented in an integrated, biologically plausible neural model. The core of this system is a model of ``flat'' RL that implements the processing of a single layer. This includes computing action values given the current state, selecting an output action based on those values, computing a temporal difference error based on the result of that action, and using that error to update the action values. We then show how the design of this system allows multiple layers to be combined hierarchically, where inputs are received from higher layers and outputs delivered to lower layers. We also provide a detailed neuroanatomical mapping, showing how the components of the model fit within known neuroanatomical structures. We demonstrate the performance of the model in a range of different environments, in order to emphasize the aim of understanding the brain's general, flexible reinforcement learning ability. These results show that the model compares well to previous modelling work and demonstrates improved performance as a result of its hierarchical ability. We also show that the model's output is consistent with available data on human hierarchical RL. Thus we believe that this work, as the first biologically plausible neural model of HRL, brings us closer to understanding the full range of RL processing in real neural systems. We conclude with a discussion of the design decisions made throughout the course of this work, as well as some of the most important avenues for the model's future development. Two of the most critical of these are the incorporation of model-based reasoning and the autonomous development of hierarchical structure, both of which are important aspects of the full HRL process that are absent in the current model. We also discuss some of the predictions that arise from this model, and how they might be tested experimentally.4 month

Estudio y realización de un neuroprocesador biológico: métodos de aprendizaje

[SPA] La presente tesis se enmarca dentro de dos campos diferentes pero intrínsecamente unidos entre sí en este caso: neurociencia y computación. El objetivo global de esta tesis es la realización de un neuroprocesador biológico empleando como plataforma redes neuronales biológicas cultivadas sobre matrices de microelectrodos. Este objetivo global resulta en una serie de sub-objetivos: (1) Definición y construcción de una plataforma para el soporte en tiempo real de los sistemas de adquisición de registros neuronales, y estimulación eléctrica de los mismos, que se comunique remotamente con un dispositivo robótico. (2) Estudio y propuesta de un método de guiado robótico basado en una plataforma de lazo cerrado que integre la información de los sensores del robot en el neuroprocesador y, en función de la respuesta de éste, direccione el sistema robótico. (3) Normalización y calibración estadística de los registros del neuroprocesador para su adecuación a los distintos algoritmos de guiado robótico y aprendizaje en los cultivos neuronales. (4) Estudio y definición de técnicas de aprendizaje en cultivos neuronales para la realización de conectividad funcional dirigida con objeto de proporcionar nuevos paradigmas de programación en neuroprocesadores biológicos. Con respecto al sub-objetivo (1), se ha propuesto un sistema de experimentación con cultivos neuronales en lazo cerrado y tiempo real que proporciona las herramientas de filtrado, visualización, procesamiento y estimulación de la respuesta electrofisiológica de poblaciones neuronales y su comunicación con un sistema robótico remoto. Para alcanzar el objetivo (2), se ha adaptado el algoritmo de centro de área para guiado robótico a las respuestas funcionales de las poblaciones de neuronas, identificando aquellos electrodos de la matriz cuyas neuronas incrementan en mayor medida sus disparos, como objetivo para el direccionamiento del robot. El cumplimiento del sub-objetivo (3) se ha conseguido al proporcionar técnicas de calibración y normalización estadística de los registros de poblaciones de neuronas que conforman el neuroprocesador, con objeto de suprimir la variabilidad intrínseca de las mismas y a las distintas características de no-homogeneidad tanto en la densidad del cultivo como en las propiedades eléctricas de los distintos electrodos. Finalmente, atendiendo al sub-objetivo (4), se ha propuesto un paradigma de aprendizaje natural, como es el aprendizaje hebbiano, para la conformación de conexiones funcionales entre electrodos que no se encontraban enlazados previamente y conseguir de esta forma el modelado del cultivo para la implementación en su estructura de las funciones a implementar, en este caso las estructuras de Braitenberg.[ENG] This thesis deals with two different fields, inherently related to each other in this case: neuroscience and computation. The overall objective of this thesis is the development of a biological neuroprocessor with cultured biological neural networks using microelectrode arrays as platform. This objective results in a set of specific subobjectives: (1) Define and build a platform for real time support of acquisition systems and electrical stimulation systems of neural registers, which remotely communicates with a robotic device. (2) Study and propose a robotic guidance method based on a close-loop platform which includes the sensory robot information in the neuroprocessor and, according to its response, guides the robotic system. (3) Normalization and statistic calibration of the registers of the neuroprocessor in order to adapt them to different algorithms of robotic guidance and learning in cultured neural networks. (4) Study and define learning techniques in neural cultures for the development of functional connectivity which allows new programming paradigms in biological neuroprocessors. Regarding objective (1), a real-time close-loop experimentation system with neural cultures has been proposed, which provides a complete solution for filtering, visualization, processing and stimulation of electrophysiological response from neural population and communication with a robotic system. In order to reach objective (2), centre of area algorithm for robotic guidance has been adapted to the functional response of neural populations, identifying those electrodes from the array whose neurons increase the most its firing rate, as target for robotic guidance. Objective (3) has been met giving statistic calibration and normalization techniques of neural population registers that conform the neuroprocessor having in mind the goal of supressing the intrinsic variability of those populations and the different nonhomogeneity characteristics, both in culture density and electrical properties of the electrodes. Finally, regarding objective (4), a natural learning paradigm has been proposed, Hebbian learning, to conform functional connections between previously not connected electrodes. In this way, the cultures can be modelled for implementing the desired behaviour in the biological structure, in this case Braitenberg behaviour.Universidad Politécnica de Cartagen

A new class of neural architectures to model episodic memory : computational studies of distal reward learning

A computational cognitive neuroscience model is proposed, which models episodic memory based on the mammalian brain. A computational neural architecture instantiates the proposed model and is tested on a particular task of distal reward learning. Categorical Neural Semantic Theory informs the architecture design. To experiment upon the computational brain model, embodiment and an environment in which the embodiment exists are simulated. This simulated environment realizes the Morris Water Maze task, a well established biological experimental test of distal reward learning. The embodied neural architecture is treated as a virtual rat and the environment it acts in as a virtual water tank. Performance levels of the neural architectures are evaluated through analysis of embodied behavior in the distal reward learning task. Comparison is made to biological rat experimental data, as well as comparison to other published models. In addition, differences in performance are compared between the normal and categorically informed versions of the architecture

2022 roadmap on neuromorphic computing and engineering

Modern computation based on von Neumann architecture is now a mature cutting-edge science. In the von Neumann architecture, processing and memory units are implemented as separate blocks interchanging data intensively and continuously. This data transfer is responsible for a large part of the power consumption. The next generation computer technology is expected to solve problems at the exascale with 10$^{18}$ calculations each second. Even though these future computers will be incredibly powerful, if they are based on von Neumann type architectures, they will consume between 20 and 30 megawatts of power and will not have intrinsic physically built-in capabilities to learn or deal with complex data as our brain does. These needs can be addressed by neuromorphic computing systems which are inspired by the biological concepts of the human brain. This new generation of computers has the potential to be used for the storage and processing of large amounts of digital information with much lower power consumption than conventional processors. Among their potential future applications, an important niche is moving the control from data centers to edge devices. The aim of this roadmap is to present a snapshot of the present state of neuromorphic technology and provide an opinion on the challenges and opportunities that the future holds in the major areas of neuromorphic technology, namely materials, devices, neuromorphic circuits, neuromorphic algorithms, applications, and ethics. The roadmap is a collection of perspectives where leading researchers in the neuromorphic community provide their own view about the current state and the future challenges for each research area. We hope that this roadmap will be a useful resource by providing a concise yet comprehensive introduction to readers outside this field, for those who are just entering the field, as well as providing future perspectives for those who are well established in the neuromorphic computing community