Deterministic networks for probabilistic computing

Neural-network models of high-level brain functions such as memory recall and reasoning often rely on the presence of stochasticity. The majority of these models assumes that each neuron in the functional network is equipped with its own private source of randomness, often in the form of uncorrelated external noise. However, both in vivo and in silico, the number of noise sources is limited due to space and bandwidth constraints. Hence, neurons in large networks usually need to share noise sources. Here, we show that the resulting shared-noise correlations can significantly impair the performance of stochastic network models. We demonstrate that this problem can be overcome by using deterministic recurrent neural networks as sources of uncorrelated noise, exploiting the decorrelating effect of inhibitory feedback. Consequently, even a single recurrent network of a few hundred neurons can serve as a natural noise source for large ensembles of functional networks, each comprising thousands of units. We successfully apply the proposed framework to a diverse set of binary-unit networks with different dimensionalities and entropies, as well as to a network reproducing handwritten digits with distinct predefined frequencies. Finally, we show that the same design transfers to functional networks of spiking neurons.Comment: 22 pages, 11 figure

Spiking neurons with short-term synaptic plasticity form superior generative networks

Spiking networks that perform probabilistic inference have been proposed both as models of cortical computation and as candidates for solving problems in machine learning. However, the evidence for spike-based computation being in any way superior to non-spiking alternatives remains scarce. We propose that short-term plasticity can provide spiking networks with distinct computational advantages compared to their classical counterparts. In this work, we use networks of leaky integrate-and-fire neurons that are trained to perform both discriminative and generative tasks in their forward and backward information processing paths, respectively. During training, the energy landscape associated with their dynamics becomes highly diverse, with deep attractor basins separated by high barriers. Classical algorithms solve this problem by employing various tempering techniques, which are both computationally demanding and require global state updates. We demonstrate how similar results can be achieved in spiking networks endowed with local short-term synaptic plasticity. Additionally, we discuss how these networks can even outperform tempering-based approaches when the training data is imbalanced. We thereby show how biologically inspired, local, spike-triggered synaptic dynamics based simply on a limited pool of synaptic resources can allow spiking networks to outperform their non-spiking relatives.Comment: corrected typo in abstrac

Stochasticity from function -- why the Bayesian brain may need no noise

An increasing body of evidence suggests that the trial-to-trial variability of spiking activity in the brain is not mere noise, but rather the reflection of a sampling-based encoding scheme for probabilistic computing. Since the precise statistical properties of neural activity are important in this context, many models assume an ad-hoc source of well-behaved, explicit noise, either on the input or on the output side of single neuron dynamics, most often assuming an independent Poisson process in either case. However, these assumptions are somewhat problematic: neighboring neurons tend to share receptive fields, rendering both their input and their output correlated; at the same time, neurons are known to behave largely deterministically, as a function of their membrane potential and conductance. We suggest that spiking neural networks may, in fact, have no need for noise to perform sampling-based Bayesian inference. We study analytically the effect of auto- and cross-correlations in functionally Bayesian spiking networks and demonstrate how their effect translates to synaptic interaction strengths, rendering them controllable through synaptic plasticity. This allows even small ensembles of interconnected deterministic spiking networks to simultaneously and co-dependently shape their output activity through learning, enabling them to perform complex Bayesian computation without any need for noise, which we demonstrate in silico, both in classical simulation and in neuromorphic emulation. These results close a gap between the abstract models and the biology of functionally Bayesian spiking networks, effectively reducing the architectural constraints imposed on physical neural substrates required to perform probabilistic computing, be they biological or artificial

Characterization and Compensation of Network-Level Anomalies in Mixed-Signal Neuromorphic Modeling Platforms

Advancing the size and complexity of neural network models leads to an ever increasing demand for computational resources for their simulation. Neuromorphic devices offer a number of advantages over conventional computing architectures, such as high emulation speed or low power consumption, but this usually comes at the price of reduced configurability and precision. In this article, we investigate the consequences of several such factors that are common to neuromorphic devices, more specifically limited hardware resources, limited parameter configurability and parameter variations. Our final aim is to provide an array of methods for coping with such inevitable distortion mechanisms. As a platform for testing our proposed strategies, we use an executable system specification (ESS) of the BrainScaleS neuromorphic system, which has been designed as a universal emulation back-end for neuroscientific modeling. We address the most essential limitations of this device in detail and study their effects on three prototypical benchmark network models within a well-defined, systematic workflow. For each network model, we start by defining quantifiable functionality measures by which we then assess the effects of typical hardware-specific distortion mechanisms, both in idealized software simulations and on the ESS. For those effects that cause unacceptable deviations from the original network dynamics, we suggest generic compensation mechanisms and demonstrate their effectiveness. Both the suggested workflow and the investigated compensation mechanisms are largely back-end independent and do not require additional hardware configurability beyond the one required to emulate the benchmark networks in the first place. We hereby provide a generic methodological environment for configurable neuromorphic devices that are targeted at emulating large-scale, functional neural networks

Accelerated physical emulation of Bayesian inference in spiking neural networks

The massively parallel nature of biological information processing plays an important role for its superiority to human-engineered computing devices. In particular, it may hold the key to overcoming the von Neumann bottleneck that limits contemporary computer architectures. Physical-model neuromorphic devices seek to replicate not only this inherent parallelism, but also aspects of its microscopic dynamics in analog circuits emulating neurons and synapses. However, these machines require network models that are not only adept at solving particular tasks, but that can also cope with the inherent imperfections of analog substrates. We present a spiking network model that performs Bayesian inference through sampling on the BrainScaleS neuromorphic platform, where we use it for generative and discriminative computations on visual data. By illustrating its functionality on this platform, we implicitly demonstrate its robustness to various substrate-specific distortive effects, as well as its accelerated capability for computation. These results showcase the advantages of brain-inspired physical computation and provide important building blocks for large-scale neuromorphic applications.Comment: This preprint has been published 2019 November 14. Please cite as: Kungl A. F. et al. (2019) Accelerated Physical Emulation of Bayesian Inference in Spiking Neural Networks. Front. Neurosci. 13:1201. doi: 10.3389/fnins.2019.0120

Cortical oscillations implement a backbone for sampling-based computation in spiking neural networks

Brains need to deal with an uncertain world. Often, this requires visiting multiple interpretations of the available information or multiple solutions to an encountered problem. This gives rise to the so-called mixing problem: since all of these "valid" states represent powerful attractors, but between themselves can be very dissimilar, switching between such states can be difficult. We propose that cortical oscillations can be effectively used to overcome this challenge. By acting as an effective temperature, background spiking activity modulates exploration. Rhythmic changes induced by cortical oscillations can then be interpreted as a form of simulated tempering. We provide a rigorous mathematical discussion of this link and study some of its phenomenological implications in computer simulations. This identifies a new computational role of cortical oscillations and connects them to various phenomena in the brain, such as sampling-based probabilistic inference, memory replay, multisensory cue combination and place cell flickering.Comment: 30 pages, 11 figure

Tuning the Magnetic Anisotropy at a Molecule-Metal Interface

International audienceWe demonstrate that a C 60 overlayer enhances the perpendicular magnetic anisotropy of a Co thin film, inducing an inverse spin reorientation transition from in plane to out of plane. The driving force is the C 60 =Co interfacial magnetic anisotropy that we have measured quantitatively in situ as a function of the C 60 coverage. Comparison with state-of-the-art ab initio calculations show that this interfacial anisotropy mainly arises from the local hybridization between C 60 p z and Co d z 2 orbitals. By generalizing these arguments, we also demonstrate that the hybridization of C 60 with a Fe(110) surface decreases the perpendicular magnetic anisotropy. These results open the way to tailor the interfacial magnetic anisotropy in organic-material–ferromagnet systems

The calcilytic agent NPS 2143 rectifies hypocalcemia in a mouse model with an activating calcium-sensing-receptor (CaSR) mutation:relevance to autosomal dominant hypocalcemia type 1 (ADH1)

Autosomal dominant hypocalcemia type 1 (ADH1) is caused by germline gain-of-function mutations of the calcium-sensing receptor (CaSR) and may lead to symptomatic hypocalcemia, inappropriately low serum parathyroid hormone (PTH) concentrations and hypercalciuria. Negative allosteric CaSR modulators, known as calcilytics, have been shown to normalise the gain-of-function associated with ADH-causing CaSR mutations in vitro and represent a potential targeted therapy for ADH1. However, the effectiveness of calcilytic drugs for the treatment of ADH1-associated hypocalcemia remains to be established. We have investigated NPS 2143, a calcilytic compound, for the treatment of ADH1 by in vitro and in vivo studies involving a mouse model, known as Nuf, which harbors a gain-of-function CaSR mutation, Leu723Gln. Wild-type (Leu723) and Nuf mutant (Gln723) CaSRs were expressed in HEK293 cells and the effect of NPS 2143 on their intracellular calcium responses determined by flow cytometry. NPS 2143 was also administered as a single intraperitoneal bolus to wild-type and Nuf mice and plasma concentrations of calcium and PTH, and urinary calcium excretion measured. In vitro administration of NPS 2143 decreased the intracellular calcium responses of HEK293 cells expressing the mutant Gln723 CaSR in a dose-dependent manner, thereby rectifying the gain-of-function associated with the Nuf mouse CaSR mutation. Intraperitoneal injection of NPS 2143 in Nuf mice led to significant increases in plasma calcium and PTH without elevating urinary calcium excretion. These studies of a mouse model with an activating CaSR mutation demonstrate NPS 2143 to normalize the gain-of-function causing ADH1, and improve the hypocalcemia associated with this disorder