8 research outputs found
Deep Learning in Neuronal and Neuromorphic Systems
The ever-increasing compute and energy requirements in the field of deep learning have caused a rising interest in the development of novel, more energy-efficient computing paradigms to support the advancement of artificial intelligence systems. Neuromorphic architectures are promising candidates, as they aim to mimic the functional mechanisms, and thereby inherit the efficiency, of their archetype: the brain. However, even though neuromorphics and deep learning are, at their roots, inspired by the brain, they are not directly compatible with each other. In this thesis, we aim at bridging this gap by realizing error backpropagation, the central algorithm behind deep learning, on neuromorphic platforms.
We start by introducing the Yin-Yang classification dataset, a tool for neuromorphic and algorithmic prototyping, as a prerequisite for the other work presented. This novel dataset is designed to not require excessive hardware or computing resources to be solved. At the same time, it is challenging enough to be useful for debugging and testing by revealing potential algorithmic or implementation flaws. We then explore two different approaches of implementing error backpropagation on neuromorphic systems. Our first solution provides an exact algorithm for error backpropagation on the first spike times of leaky integrate-andfire neurons, one of the most common neuron models implemented in neuromorphic chips. The neuromorphic feasibility is demonstrated by the deployment on the BrainScaleS-2 chip and yields competitive results both with respect to task performance as well as efficiency. The second approach is based on a biologically plausible variant of error backpropagation realized by a dendritc microcircuit model. We assess this model with respect to its practical feasibility, extend it to improve learning performance and address the obstacles for neuromorphic implementation: We introduce the Latent Equilibrium mechanism to solve the relaxation problem introduced by slow neuron dynamics. Our Phaseless Alignment Learning method allows us to learn feedback weights in the network and thus avoid the weight transport problem. And finally, we explore two methods to port the rate-based model onto an event-based neuromorphic system.
The presented work showcases two ways of uniting the powerful and flexible learning mechanisms of deep learning with energy-efficient neuromorphic systems, thus illustrating the potential of a convergence of artificial intelligence and neuromorphic engineering research
The Yin-Yang dataset
The Yin-Yang dataset was developed for research on biologically plausible error backpropagation and deep learning in spiking neural networks. It serves as an alternative to classic deep learning datasets, especially in early-stage prototyping scenarios for both network models and hardware platforms, for which it provides several advantages. First, it is smaller and therefore faster to learn, thereby being better suited for small-scale exploratory studies in both software simulations and hardware prototypes. Second, it exhibits a very clear gap between the accuracies achievable using shallow as compared to deep neural networks. Third, it is easily transferable between spatial and temporal input domains, making it interesting for different types of classification scenarios
Gradient-based methods for spiking physical systems
Recent efforts have fostered significant progress towards deep learning in
spiking networks, both theoretical and in silico. Here, we discuss several
different approaches, including a tentative comparison of the results on
BrainScaleS-2, and hint towards future such comparative studies.Comment: 2 page abstract, submitted to and accepted by the NNPC (International
conference on neuromorphic, natural and physical computing
Fast and deep: energy-efficient neuromorphic learning with first-spike times
For a biological agent operating under environmental pressure, energy
consumption and reaction times are of critical importance. Similarly,
engineered systems also strive for short time-to-solution and low
energy-to-solution characteristics. At the level of neuronal implementation,
this implies achieving the desired results with as few and as early spikes as
possible. In the time-to-first-spike-coding framework, both of these goals are
inherently emerging features of learning. Here, we describe a rigorous
derivation of learning such first-spike times in networks of leaky
integrate-and-fire neurons, relying solely on input and output spike times, and
show how it can implement error backpropagation in hierarchical spiking
networks. Furthermore, we emulate our framework on the BrainScaleS-2
neuromorphic system and demonstrate its capability of harnessing the chip's
speed and energy characteristics. Finally, we examine how our approach
generalizes to other neuromorphic platforms by studying how its performance is
affected by typical distortive effects induced by neuromorphic substrates.Comment: 20 pages, 8 figure
NeuroBench:Advancing Neuromorphic Computing through Collaborative, Fair and Representative Benchmarking
The field of neuromorphic computing holds great promise in terms of advancing computing efficiency and capabilities by following brain-inspired principles. However, the rich diversity of techniques employed in neuromorphic research has resulted in a lack of clear standards for benchmarking, hindering effective evaluation of the advantages and strengths of neuromorphic methods compared to traditional deep-learning-based methods. This paper presents a collaborative effort, bringing together members from academia and the industry, to define benchmarks for neuromorphic computing: NeuroBench. The goals of NeuroBench are to be a collaborative, fair, and representative benchmark suite developed by the community, for the community. In this paper, we discuss the challenges associated with benchmarking neuromorphic solutions, and outline the key features of NeuroBench. We believe that NeuroBench will be a significant step towards defining standards that can unify the goals of neuromorphic computing and drive its technological progress. Please visit neurobench.ai for the latest updates on the benchmark tasks and metrics
NeuroBench: Advancing Neuromorphic Computing through Collaborative, Fair and Representative Benchmarking
The field of neuromorphic computing holds great promise in terms of advancing
computing efficiency and capabilities by following brain-inspired principles.
However, the rich diversity of techniques employed in neuromorphic research has
resulted in a lack of clear standards for benchmarking, hindering effective
evaluation of the advantages and strengths of neuromorphic methods compared to
traditional deep-learning-based methods. This paper presents a collaborative
effort, bringing together members from academia and the industry, to define
benchmarks for neuromorphic computing: NeuroBench. The goals of NeuroBench are
to be a collaborative, fair, and representative benchmark suite developed by
the community, for the community. In this paper, we discuss the challenges
associated with benchmarking neuromorphic solutions, and outline the key
features of NeuroBench. We believe that NeuroBench will be a significant step
towards defining standards that can unify the goals of neuromorphic computing
and drive its technological progress. Please visit neurobench.ai for the latest
updates on the benchmark tasks and metrics
Latent Equilibrium: A unified learning theory for arbitrarily fast computation with arbitrarily slow neurons
The response time of physical computational elements is finite, and neurons are no exception. In hierarchical models of cortical networks each layer thus introduces a response lag. This inherent property of physical dynamical systems results in delayed processing of stimuli and causes a timing mismatch between network output and instructive signals, thus afflicting not only inference, but also learning. We introduce Latent Equilibrium, a new framework for inference and learning in networks of slow components which avoids these issues by harnessing the ability of biological neurons to phase-advance their output with respect to their membrane potential. This principle enables quasi-instantaneous inference independent of network depth and avoids the need for phased plasticity or computationally expensive network relaxation phases. We jointly derive disentangled neuron and synapse dynamics from a prospective energy function that depends on a network's generalized position and momentum. The resulting model can be interpreted as a biologically plausible approximation of error backpropagation in deep cortical networks with continuous-time, leaky neuronal dynamics and continuously active, local plasticity. We demonstrate successful learning of standard benchmark datasets, achieving competitive performance using both fully-connected and convolutional architectures, and show how our principle can be applied to detailed models of cortical microcircuitry. Furthermore, we study the robustness of our model to spatio-temporal substrate imperfections to demonstrate its feasibility for physical realization, be it in vivo or in silico