Autonomous Navigation Using Reinforcement Learning with Spiking Neural Networks

The autonomous navigation of mobile robots is of great interest in mobile robotics. Algorithms such as simultaneous localization and mapping (SLAM) and artificial potential field methods can be applied to known and mapped environments. However, navigating in an unknown, and unmapped environments is still a challenge. In this research, we propose an algorithm for mobile robot navigation in the near-shortest possible time toward a predefined target location in an unknown environment containing obstacles. The algorithm is based on a reinforcement learning paradigm with biologically realistic spiking neural networks. We make use of eligibility traces that are inherent to spiking neural networks to solve the delayed reward problem implicitly present in reinforcement learning. With this algorithm, we achieve a set of movement decisions for the mobile robot to reach the target in the near-shortest time

Gridbot: An autonomous robot controlled by a Spiking Neural Network mimicking the brain's navigational system

It is true that the "best" neural network is not necessarily the one with the most "brain-like" behavior. Understanding biological intelligence, however, is a fundamental goal for several distinct disciplines. Translating our understanding of intelligence to machines is a fundamental problem in robotics. Propelled by new advancements in Neuroscience, we developed a spiking neural network (SNN) that draws from mounting experimental evidence that a number of individual neurons is associated with spatial navigation. By following the brain's structure, our model assumes no initial all-to-all connectivity, which could inhibit its translation to a neuromorphic hardware, and learns an uncharted territory by mapping its identified components into a limited number of neural representations, through spike-timing dependent plasticity (STDP). In our ongoing effort to employ a bioinspired SNN-controlled robot to real-world spatial mapping applications, we demonstrate here how an SNN may robustly control an autonomous robot in mapping and exploring an unknown environment, while compensating for its own intrinsic hardware imperfections, such as partial or total loss of visual input.Comment: 8 pages, 3 Figures, International Conference on Neuromorphic Systems (ICONS 2018

Biologically Inspired Dynamic Thresholds for Spiking Neural Networks

The dynamic membrane potential threshold, as one of the essential properties of a biological neuron, is a spontaneous regulation mechanism that maintains neuronal homeostasis, i.e., the constant overall spiking firing rate of a neuron. As such, the neuron firing rate is regulated by a dynamic spiking threshold, which has been extensively studied in biology. Existing work in the machine learning community does not employ bioinspired spiking threshold schemes. This work aims at bridging this gap by introducing a novel bioinspired dynamic energy-temporal threshold (BDETT) scheme for spiking neural networks (SNNs). The proposed BDETT scheme mirrors two bioplausible observations: a dynamic threshold has 1) a positive correlation with the average membrane potential and 2) a negative correlation with the preceding rate of depolarization. We validate the effectiveness of the proposed BDETT on robot obstacle avoidance and continuous control tasks under both normal conditions and various degraded conditions, including noisy observations, weights, and dynamic environments. We find that the BDETT outperforms existing static and heuristic threshold approaches by significant margins in all tested conditions, and we confirm that the proposed bioinspired dynamic threshold scheme offers homeostasis to SNNs in complex real-world tasks

Supervised Learning in SNN via Reward-Modulated Spike-Timing-Dependent Plasticity for a Target Reaching Vehicle

Spiking neural networks (SNNs) offer many advantages over traditional artificial neural networks (ANNs) such as biological plausibility, fast information processing, and energy efficiency. Although SNNs have been used to solve a variety of control tasks using the Spike-Timing-Dependent Plasticity (STDP) learning rule, existing solutions usually involve hard-coded network architectures solving specific tasks rather than solving different kinds of tasks generally. This results in neglecting one of the biggest advantages of ANNs, i.e., being general-purpose and easy-to-use due to their simple network architecture, which usually consists of an input layer, one or multiple hidden layers and an output layer. This paper addresses the problem by introducing an end-to-end learning approach of spiking neural networks constructed with one hidden layer and reward-modulated Spike-Timing-Dependent Plasticity (R-STDP) synapses in an all-to-all fashion. We use the supervised reward-modulated Spike-Timing-Dependent-Plasticity learning rule to train two different SNN-based sub-controllers to replicate a desired obstacle avoiding and goal approaching behavior, provided by pre-generated datasets. Together they make up a target-reaching controller, which is used to control a simulated mobile robot to reach a target area while avoiding obstacles in its path. We demonstrate the performance and effectiveness of our trained SNNs to achieve target reaching tasks in different unknown scenarios

Tuning Synaptic Connections instead of Weights by Genetic Algorithm in Spiking Policy Network

Learning from the interaction is the primary way biological agents know about the environment and themselves. Modern deep reinforcement learning (DRL) explores a computational approach to learning from interaction and has significantly progressed in solving various tasks. However, the powerful DRL is still far from biological agents in energy efficiency. Although the underlying mechanisms are not fully understood, we believe that the integration of spiking communication between neurons and biologically-plausible synaptic plasticity plays a prominent role. Following this biological intuition, we optimize a spiking policy network (SPN) by a genetic algorithm as an energy-efficient alternative to DRL. Our SPN mimics the sensorimotor neuron pathway of insects and communicates through event-based spikes. Inspired by biological research that the brain forms memories by forming new synaptic connections and rewires these connections based on new experiences, we tune the synaptic connections instead of weights in SPN to solve given tasks. Experimental results on several robotic control tasks show that our method can achieve the performance level of mainstream DRL methods and exhibit significantly higher energy efficiency

Analysis and Development of Computational Intelligence based Navigational Controllers for Multiple Mobile Robots

Navigational path planning problems of the mobile robots have received considerable attention over the past few decades. The navigation problem of mobile robots are consisting of following three aspects i.e. locomotion, path planning and map building. Based on these three aspects path planning algorithm for a mobile robot is formulated, which is capable of finding an optimal collision free path from the start point to the target point in a given environment. The main objective of the dissertation is to investigate the advanced methodologies for both single and multiple mobile robots navigation in highly cluttered environments using computational intelligence approach. Firstly, three different standalone computational intelligence approaches based on the Adaptive Neuro-Fuzzy Inference System (ANFIS), Cuckoo Search (CS) algorithm and Invasive Weed Optimization (IWO) are presented to address the problem of path planning in unknown environments. Next two different hybrid approaches are developed using CS-ANFIS and IWO-ANFIS to solve the mobile robot navigation problems. The performance of each intelligent navigational controller is demonstrated through simulation results using MATLAB. Experimental results are conducted in the laboratory, using real mobile robots to validate the versatility and effectiveness of the proposed navigation techniques. Comparison studies show, that there are good agreement between them. During the analysis of results, it is noticed that CS-ANFIS and IWO-ANFIS hybrid navigational controllers perform better compared to other discussed navigational controllers. The results obtained from the proposed navigation techniques are validated by comparison with the results from other intelligent techniques such as Fuzzy logic, Neural Network, Genetic Algorithm (GA), Particle Swarm Optimization (PSO), Ant Colony Optimization (ACO) and other hybrid algorithms. By investigating the results, finally it is concluded that the proposed navigational methodologies are efficient and robust in the sense, that they can be effectively implemented to solve the path optimization problems of mobile robot in any complex environment

Navigational Path Analysis of Mobile Robot in Various Environments

This dissertation describes work in the area of an autonomous mobile robot. The objective is navigation of mobile robot in a real world dynamic environment avoiding structured and unstructured obstacles either they are static or dynamic. The shapes and position of obstacles are not known to robot prior to navigation. The mobile robot has sensory recognition of specific objects in the environments. This sensory-information provides local information of robots immediate surroundings to its controllers. The information is dealt intelligently by the robot to reach the global objective (the target). Navigational paths as well as time taken during navigation by the mobile robot can be expressed as an optimisation problem and thus can be analyzed and solved using AI techniques. The optimisation of path as well as time taken is based on the kinematic stability and the intelligence of the robot controller. A successful way of structuring the navigation task deals with the issues of individual behaviour design and action coordination of the behaviours. The navigation objective is addressed using fuzzy logic, neural network, adaptive neuro-fuzzy inference system and different other AI technique.The research also addresses distributed autonomous systems using multiple robot

Analysis and Control of Mobile Robots in Various Environmental Conditions

The world sees new inventions each day, made to make the lifestyle of humans more easy and luxurious. In such global scenario, the robots have proved themselves to be an invention of great importance. The robots are being used in almost each and every field of the human world. Continuous studies are being done on them to make them simpler and easier to work with. All fields are being unraveled to make them work better in the human world without human interference. We focus on the navigation field of these mobile robots. The aim of this thesis is to find the controller that produces the most optimal path for the robot to reach its destination without colliding or damaging itself or the environment. The techniques like Fuzzy logic, Type 2 fuzzy logic, Neural networks and Artificial bee colony have been discussed and experimented to find the best controller that could find the most optimal path for the robot to reach its goal position. Simulation and Experiments have been done alike to find out the optimal path for the robot

Reservoir Computing Architectures for Modeling Robot Navigation Systems

This thesis proposes a new efficient and biologically inspired way of modeling navigation tasks for autonomous mobile robots having restrictions on cost, energy consumption, and computational complexity (such as household and assistant robots). It is based on the recently proposed Reservoir Computing approach for training Recurrent Neural Networks. Robot Navigation Systems Autonomous mobile robots must be able to safely and purposefully navigate in complex dynamic environments, preferentially considering a restricted amount of computational power as well as limited energy consumption. In order to turn these robots into commercially viable domestic products with intelligent, abstract computational capabilities, it is also necessary to use inexpensive sensory apparatus such as a few infra-red distance sensors of limited accuracy. Current state-of-the-art methods for robot localization and navigation require fully equipped robotic platforms usually possessing expensive laser scanners for environment mapping, a considerable amount of computational power, and extensive explicit modeling of the environment and of the task. This thesis The research presented in this thesis is a step towards creating intelligent autonomous mobile robots with abstract reasoning capabilities using a limited number of very simple raw noisy sensory signals, such as distance sensors. The basic assumption is that the low-dimensional sensory signal can be projected into a high-dimensional dynamic space where learning and computation is performed by linear methods (such as linear regression), overcoming sensor aliasing problems commonly found in robot navigation tasks. This form of computation is known in the literature as Reservoir Computing (RC), and the Echo State Network is a particular RC model used in this work and characterized by having the high-dimensional space implemented by a discrete analog recurrent neural network with fading memory properties. This thesis proposes a number of Reservoir Computing architectures which can be used in a variety of autonomous navigation tasks, by modeling implicit abstract representations of an environment as well as navigation behaviors which can be sequentially executed in the physical environment or simulated as a plan in deliberative goal-directed tasks. Navigation attractors A navigation attractor is a reactive robot behavior defined by a temporal pattern of sensory-motor coupling through the environment space. Under this scheme, a robot tends to follow a trajectory with attractor-like characteristics in space. These navigation attractors are characterized by being robust to noise and unpredictable events and by having inherent collision avoidance skills. In this work, it is shown that an RC network can model not only one behavior, but multiple navigation behaviors by shifting the operating point of the dynamical reservoir system into different \emph{sub-space attractors} using additional external inputs representing the selected behavior. The sub-space attractors emerge from the coupling existing between the RC network, which controls the autonomous robot, and the environment. All this is achieved under an imitation learning framework which trains the RC network using examples of navigation behaviors generated by a supervisor controller or a human. Implicit spatial representations From the stream of sensory input given by distance sensors, it is possible to construct implicit spatial representations of an environment by using Reservoir Computing networks. These networks are trained in a supervised way to predict locations at different levels of abstraction, from continuous-valued robot's pose in the global coordinate's frame, to more abstract locations such as small delimited areas and rooms of a robot environment. The high-dimensional reservoir projects the sensory input into a dynamic system space, whose characteristic fading memory disambiguates the sensory space, solving the sensor aliasing problems where multiple different locations generate similar sensory readings from the robot's perspective. Hierarchical networks for goal-directed navigation It is possible to model navigation attractors and implicit spatial representations with the same type of RC network. By constructing an hierarchical RC architecture which combines the aforementioned modeling skills in two different reservoir modules operating at different timescales, it is possible to achieve complex context-dependent sensory-motor coupling in unknown environments. The general idea is that the network trained to predict the location and orientation of the robot in this architecture can be used to select appropriate navigation attractors according to the current context, by shifting the operating point of the navigation reservoir to a sub-space attractor. As the robot navigates from one room to the next, a corresponding context switch selects a new reactive navigation behavior. This continuous sequence of context switches and reactive behaviors, when combined with an external input indicating the destination room, leads ultimately to a goal-directed navigation system, purely trained in a supervised way with examples of sensory-motor coupling. Generative modeling of environment-robot dynamics RC networks trained to predict the position of the robot from the sensory signals learns forward models of the robot. By using a generative RC network which predicts not only locations but also sensory nodes, it is possible to use the network in the opposite direction for predicting local environmental sensory perceptions from the robot position as input, thus learning an inverse model. The implicit map learned by forward models can be made explicit, by running the RC network in reverse: predict the local sensory signals given the location of the robot as input (inverse model). which are fed back to the reservoir, it is possible to internally predict future scenarios and behaviors without actually experiencing them in the current environment (a process analogous to dreaming), constituting a planning-like capability which opens new possibilities for deliberative navigation systems. Unsupervised learning of spatial representations In order to achieve a higher degree of autonomy in the learning process of RC-based navigation systems which use implicit learned models of the environment for goal-directed navigation, a new architecture is proposed. Instead of using linear regression, an unsupervised learning method which extracts slowly-varying output signals from the reservoir states, called Slow Feature Analysis, is used to generate self-organized spatial representations at the output layer, without the requirement of labeling training data with the desired locations. It is shown experimentally that the proposed RC-SFA architecture is empowered with an unique combination of short-term memory and non-linear transformations which overcomes the hidden state problem present in robot navigation tasks. In addition, experiments with simulated and real robots indicate that spatial activations generated by the trained network show similarities to the activations of CA1 hippocampal cells of rats (a specific group of neurons in the hippocampus)

Event-Driven Technologies for Reactive Motion Planning: Neuromorphic Stereo Vision and Robot Path Planning and Their Application on Parallel Hardware

Die Robotik wird immer mehr zu einem Schlüsselfaktor des technischen Aufschwungs. Trotz beeindruckender Fortschritte in den letzten Jahrzehnten, übertreffen Gehirne von Säugetieren in den Bereichen Sehen und Bewegungsplanung noch immer selbst die leistungsfähigsten Maschinen. Industrieroboter sind sehr schnell und präzise, aber ihre Planungsalgorithmen sind in hochdynamischen Umgebungen, wie sie für die Mensch-Roboter-Kollaboration (MRK) erforderlich sind, nicht leistungsfähig genug. Ohne schnelle und adaptive Bewegungsplanung kann sichere MRK nicht garantiert werden. Neuromorphe Technologien, einschließlich visueller Sensoren und Hardware-Chips, arbeiten asynchron und verarbeiten so raum-zeitliche Informationen sehr effizient. Insbesondere ereignisbasierte visuelle Sensoren sind konventionellen, synchronen Kameras bei vielen Anwendungen bereits überlegen. Daher haben ereignisbasierte Methoden ein großes Potenzial, schnellere und energieeffizientere Algorithmen zur Bewegungssteuerung in der MRK zu ermöglichen. In dieser Arbeit wird ein Ansatz zur flexiblen reaktiven Bewegungssteuerung eines Roboterarms vorgestellt. Dabei wird die Exterozeption durch ereignisbasiertes Stereosehen erreicht und die Pfadplanung ist in einer neuronalen Repräsentation des Konfigurationsraums implementiert. Die Multiview-3D-Rekonstruktion wird durch eine qualitative Analyse in Simulation evaluiert und auf ein Stereo-System ereignisbasierter Kameras übertragen. Zur Evaluierung der reaktiven kollisionsfreien Online-Planung wird ein Demonstrator mit einem industriellen Roboter genutzt. Dieser wird auch für eine vergleichende Studie zu sample-basierten Planern verwendet. Ergänzt wird dies durch einen Benchmark von parallelen Hardwarelösungen wozu als Testszenario Bahnplanung in der Robotik gewählt wurde. Die Ergebnisse zeigen, dass die vorgeschlagenen neuronalen Lösungen einen effektiven Weg zur Realisierung einer Robotersteuerung für dynamische Szenarien darstellen. Diese Arbeit schafft eine Grundlage für neuronale Lösungen bei adaptiven Fertigungsprozesse, auch in Zusammenarbeit mit dem Menschen, ohne Einbußen bei Geschwindigkeit und Sicherheit. Damit ebnet sie den Weg für die Integration von dem Gehirn nachempfundener Hardware und Algorithmen in die Industrierobotik und MRK