Comparative evaluation of approaches in T.4.1-4.3 and working definition of adaptive module

The goal of this deliverable is two-fold: (1) to present and compare different approaches towards learning and encoding movements us- ing dynamical systems that have been developed by the AMARSi partners (in the past during the first 6 months of the project), and (2) to analyze their suitability to be used as adaptive modules, i.e. as building blocks for the complete architecture that will be devel- oped in the project. The document presents a total of eight approaches, in two groups: modules for discrete movements (i.e. with a clear goal where the movement stops) and for rhythmic movements (i.e. which exhibit periodicity). The basic formulation of each approach is presented together with some illustrative simulation results. Key character- istics such as the type of dynamical behavior, learning algorithm, generalization properties, stability analysis are then discussed for each approach. We then make a comparative analysis of the different approaches by comparing these characteristics and discussing their suitability for the AMARSi project

Analog Spiking Neuromorphic Circuits and Systems for Brain- and Nanotechnology-Inspired Cognitive Computing

Human society is now facing grand challenges to satisfy the growing demand for computing power, at the same time, sustain energy consumption. By the end of CMOS technology scaling, innovations are required to tackle the challenges in a radically different way. Inspired by the emerging understanding of the computing occurring in a brain and nanotechnology-enabled biological plausible synaptic plasticity, neuromorphic computing architectures are being investigated. Such a neuromorphic chip that combines CMOS analog spiking neurons and nanoscale resistive random-access memory (RRAM) using as electronics synapses can provide massive neural network parallelism, high density and online learning capability, and hence, paves the path towards a promising solution to future energy-efficient real-time computing systems. However, existing silicon neuron approaches are designed to faithfully reproduce biological neuron dynamics, and hence they are incompatible with the RRAM synapses, or require extensive peripheral circuitry to modulate a synapse, and are thus deficient in learning capability. As a result, they eliminate most of the density advantages gained by the adoption of nanoscale devices, and fail to realize a functional computing system. This dissertation describes novel hardware architectures and neuron circuit designs that synergistically assemble the fundamental and significant elements for brain-inspired computing. Versatile CMOS spiking neurons that combine integrate-and-fire, passive dense RRAM synapses drive capability, dynamic biasing for adaptive power consumption, in situ spike-timing dependent plasticity (STDP) and competitive learning in compact integrated circuit modules are presented. Real-world pattern learning and recognition tasks using the proposed architecture were demonstrated with circuit-level simulations. A test chip was implemented and fabricated to verify the proposed CMOS neuron and hardware architecture, and the subsequent chip measurement results successfully proved the idea. The work described in this dissertation realizes a key building block for large-scale integration of spiking neural network hardware, and then, serves as a step-stone for the building of next-generation energy-efficient brain-inspired cognitive computing systems

Unmasking Clever Hans Predictors and Assessing What Machines Really Learn

Current learning machines have successfully solved hard application problems, reaching high accuracy and displaying seemingly "intelligent" behavior. Here we apply recent techniques for explaining decisions of state-of-the-art learning machines and analyze various tasks from computer vision and arcade games. This showcases a spectrum of problem-solving behaviors ranging from naive and short-sighted, to well-informed and strategic. We observe that standard performance evaluation metrics can be oblivious to distinguishing these diverse problem solving behaviors. Furthermore, we propose our semi-automated Spectral Relevance Analysis that provides a practically effective way of characterizing and validating the behavior of nonlinear learning machines. This helps to assess whether a learned model indeed delivers reliably for the problem that it was conceived for. Furthermore, our work intends to add a voice of caution to the ongoing excitement about machine intelligence and pledges to evaluate and judge some of these recent successes in a more nuanced manner.Comment: Accepted for publication in Nature Communication

Evolving robots: from simple behaviours to complete systems

Building robots is generally considered difficult, because the designer not only has to predict the interaction between the robot and the environment, but also has to deal with the ensuing problems. This thesis examines the use of the evolutionary approach in designing robots; the explorations range from evolving simple behaviours for real robots, to complex behaviours (also for real robots), and finally to complete robot systems — including controllers and body plans. A framework is presented for evolving robot control systems. It includes two components: a task independent Genetic Programming sub-system and a task dependent controller evaluation sub-system. The performance evaluation of each robot controller is done in a simulator to reduce the evaluation time, and then the evolved controllers are downloaded to a real robot for performance verification. In addition, a special rep¬ resentation is designed for the reactive robot controller. It is succinct and can capture the important characteristics of a reactive control system, so that the evolutionary system can efficiently evolve the controllers of the desired behaviours for the robots. The framework has been successfully used to evolve controllers for real robots to achieve a variety of simple tasks, such as obstacle avoidance, safe exploration and box-pushing. A methodology is then proposed to scale up the system to evolve controllers for more complicated tasks. It involves adopting the architecture of a behaviour-based system, and evolving separate behaviour controllers and arbitrators for coordination. This allows robot controllers for more complex skills to be constructed in an incremental manner. Therefore the whole control system becomes easy to evolve; moreover, the resulting control system can be explicitly distributed, understandable to the system designer, and easy to maintain. The methodology has been used to evolve control systems for more complex tasks with good results. Finally, the evolutionary mechanism of the framework described above is extended to include a Genetic Algorithm sub-system for the co-evolution of robot body plans — structuralparametersofphysicalrobotsencodedaslinearstringsofrealnumbers. An individual in the extended system thus consists of a brain(controller) and a body. Whenever the individual is evaluated, the controller is executed on the corresponding body for a period of time to measure the performance. In such a system the Genetic Programming part evolves the controller; and the Genetic Algorithm part, the robot body. The results show that the complete robot system can be evolved in this manner. i

All-Optical Spiking Neuron Based On Passive Micro-Resonator

Neuromorphic photonics that aims to process and store information simultaneously like human brains has emerged as a promising alternative for the next generation intelligent computing systems. The implementation of hardware emulating the basic functionality of neurons and synapses is the fundamental work in this field. However, previously proposed optical neurons implemented with SOA-MZIs, modulators, lasers or phase change materials are all dependent on active devices and quite difficult for integration. Meanwhile, although the nonlinearity in nanocavities has long been of interest, the previous theories are intended for specific situations, e.g., self-pulsation in microrings, and there is still a lack of systematic studies in the excitability behavior of the nanocavities including the silicon photonic crystal cavities. Here, we report for the first time a universal coupled mode theory model for all side-coupled passive microresonators. Attributed to the nonlinear excitability, the passive microresonator can function as a new type of all-optical spiking neuron. We demonstrate the microresonator-based neuron can exhibit the three most important characteristics of spiking neurons: excitability threshold, refractory period and cascadability behavior, paving the way to realize all-optical spiking neural networks.Comment: 8 pages, 7 figure

The role of reward signal in deep reinforcement learning

The goal of the thesis is to study the role of the reward signal in deep reinforcement learning. The reward signal is a scalar quantity received by the agent, and it has a big impact on both the training process of a reinforcement learning algorithm and its resulting behaviour. Firstly, we study the behaviour of an agent that is learning with different reward signals in the same environment with the same learning algorithm. We introduce and measure agents’ happiness as a relation between agents’ actual reward obtained from the environment, as compared to the possible maximum and minimum rewards in a given setting. The experiments show that the rewards intended to result in a given behaviour during training do not result in the same behaviour when agents interact with each other. Secondly, we use these observations to investigate the role of the reward signal further. Namely, we explore the space of all possible reward signals in a given environment through an evolutionary algorithm. Through experiments, we demonstrate that it is possible to learn complex behaviours of winning, losing, and cooperating through reward signal evolution. Some of the solutions found by the algorithm are surprising, in the sense that they would probably not have been chosen by a person trying to hand-code a given behaviour through a specific reward signal. The results presented in the thesis indicate that the role of the reward signal in reinforcement learning is likely bigger than indicated by its current coverage in the literature and is worth investigating in greater detail. Not only can it lead to programmes with less overfitting, but it can also improve our understanding of what reinforcement learning algorithms are really learning. This in turn will give us more robust, explainable, and overall safer systems