Mathematical properties of neuronal TD-rules and differential Hebbian learning: a comparison

A confusingly wide variety of temporally asymmetric learning rules exists related to reinforcement learning and/or to spike-timing dependent plasticity, many of which look exceedingly similar, while displaying strongly different behavior. These rules often find their use in control tasks, for example in robotics and for this rigorous convergence and numerical stability is required. The goal of this article is to review these rules and compare them to provide a better overview over their different properties. Two main classes will be discussed: temporal difference (TD) rules and correlation based (differential hebbian) rules and some transition cases. In general we will focus on neuronal implementations with changeable synaptic weights and a time-continuous representation of activity. In a machine learning (non-neuronal) context, for TD-learning a solid mathematical theory has existed since several years. This can partly be transfered to a neuronal framework, too. On the other hand, only now a more complete theory has also emerged for differential Hebb rules. In general rules differ by their convergence conditions and their numerical stability, which can lead to very undesirable behavior, when wanting to apply them. For TD, convergence can be enforced with a certain output condition assuring that the δ-error drops on average to zero (output control). Correlation based rules, on the other hand, converge when one input drops to zero (input control). Temporally asymmetric learning rules treat situations where incoming stimuli follow each other in time. Thus, it is necessary to remember the first stimulus to be able to relate it to the later occurring second one. To this end different types of so-called eligibility traces are being used by these two different types of rules. This aspect leads again to different properties of TD and differential Hebbian learning as discussed here. Thus, this paper, while also presenting several novel mathematical results, is mainly meant to provide a road map through the different neuronally emulated temporal asymmetrical learning rules and their behavior to provide some guidance for possible applications

Improved ground-state modulation characteristics in 1.3 μm InAs/GaAs quantum dot lasers by rapid thermal annealing

We investigated the ground-state (GS) modulation characteristics of 1.3 μm InAs/GaAs quantum dot (QD) lasers that consist of either as-grown or annealed QDs. The choice of annealing conditions was determined from our recently reported results. With reference to the as-grown QD lasers, one obtains approximately 18% improvement in the modulation bandwidth from the annealed QD lasers. In addition, the modulation efficiency of the annealed QD lasers improves by approximately 45% as compared to the as-grown ones. The observed improvements are due to (1) the removal of defects which act as nonradiative recombination centers in the QD structure and (2) the reduction in the Auger-related recombination processes upon annealing

Echo State Property of Deep Reservoir Computing Networks

In the last years, the Reservoir Computing (RC) framework has emerged as a state of-the-art approach for efficient learning in temporal domains. Recently, within the RC context, deep Echo State Network (ESN) models have been proposed. Being composed of a stack of multiple non-linear reservoir layers, deep ESNs potentially allow to exploit the advantages of a hierarchical temporal feature representation at different levels of abstraction, at the same time preserving the training efficiency typical of the RC methodology. In this paper, we generalize to the case of deep architectures the fundamental RC conditions related to the Echo State Property (ESP), based on the study of stability and contractivity of the resulting dynamical system. Besides providing a necessary condition and a sufficient condition for the ESP of layered RC networks, the results of our analysis provide also insights on the nature of the state dynamics in hierarchically organized recurrent models. In particular, we find out that by adding layers to a deep reservoir architecture, the regime of network’s dynamics can only be driven towards (equally or) less stable behaviors. Moreover, our investigation shows the intrinsic ability of temporal dynamics differentiation at the different levels in a deep recurrent architecture, with higher layers in the stack characterized by less contractive dynamics. Such theoretical insights are further supported by experimental results that show the effect of layering in terms of a progressively increased short-term memory capacity of the recurrent models

Tuberculosis Incidence in Prisons: A Systematic Review

A systematic review by Iacopo Baussano and colleagues synthesizes published research to show that improved tuberculosis (TB) control in prisons could significantly reduce the burden of TB both inside and outside prisons

Spike-Based Reinforcement Learning in Continuous State and Action Space: When Policy Gradient Methods Fail

Changes of synaptic connections between neurons are thought to be the physiological basis of learning. These changes can be gated by neuromodulators that encode the presence of reward. We study a family of reward-modulated synaptic learning rules for spiking neurons on a learning task in continuous space inspired by the Morris Water maze. The synaptic update rule modifies the release probability of synaptic transmission and depends on the timing of presynaptic spike arrival, postsynaptic action potentials, as well as the membrane potential of the postsynaptic neuron. The family of learning rules includes an optimal rule derived from policy gradient methods as well as reward modulated Hebbian learning. The synaptic update rule is implemented in a population of spiking neurons using a network architecture that combines feedforward input with lateral connections. Actions are represented by a population of hypothetical action cells with strong mexican-hat connectivity and are read out at theta frequency. We show that in this architecture, a standard policy gradient rule fails to solve the Morris watermaze task, whereas a variant with a Hebbian bias can learn the task within 20 trials, consistent with experiments. This result does not depend on implementation details such as the size of the neuronal populations. Our theoretical approach shows how learning new behaviors can be linked to reward-modulated plasticity at the level of single synapses and makes predictions about the voltage and spike-timing dependence of synaptic plasticity and the influence of neuromodulators such as dopamine. It is an important step towards connecting formal theories of reinforcement learning with neuronal and synaptic properties

A Novel Diagnostic and Prognostic Score for Abdominal Aortic Aneurysms Based on D-Dimer and a Comprehensive Analysis of Myeloid Cell Parameters

The pathogenesis of abdominal aortic aneurysm (AAA) involves a central component of chronic inflammation which is predominantly mediated by myeloid cells. We hypothesized that the local inflammatory activity may be reflected in systemic alterations of neutrophil and monocyte populations as well as in soluble factors of myeloid cell activation and recruitment. To establish their marker potential, neutrophil and monocyte sub-sets were measured by flow cytometry in peripheral blood samples of 41 AAA patients and 38 healthy controls matched for age, sex, body mass index and smoking habit. Comparably, circulating factors reflecting neutrophil and monocyte activation and recruitment were assayed in plasma. Significantly elevated levels of CD16+ monocytes, activated neutrophils and newly released neutrophils were recorded for AAA patients compared with controls. In line, the monocyte chemoattractant C-C chemokine ligand 2 and myeloperoxidase were significantly increased in patients' plasma. The diagnostic value was highest for myeloperoxidase, a mediator which is released by activated neutrophils as well as CD16+ monocytes. Multivariable regression models using myeloid activation markers and routine laboratory parameters identified myeloperoxidase and D-dimer as strong independent correlates of AAA. These two biomarkers were combined to yield a diagnostic score which was subsequently challenged for confounders and confirmed in a validation cohort matched for cardiovascular disease. Importantly, the score was also found suited to predict rapid disease progression. In conclusion, D-dimer and myeloperoxidase represent two sensitive biomarkers of AAA which reflect distinct hallmarks (thrombus formation and inflammation) of the pathomechanism and, when combined, may serve as diagnostic and prognostic AAA score warranting further evaluation

A Barcode Screen for Epigenetic Regulators Reveals a Role for the NuB4/HAT-B Histone Acetyltransferase Complex in Histone Turnover

Dynamic modification of histone proteins plays a key role in regulating gene expression. However, histones themselves can also be dynamic, which potentially affects the stability of histone modifications. To determine the molecular mechanisms of histone turnover, we developed a parallel screening method for epigenetic regulators by analyzing chromatin states on DNA barcodes. Histone turnover was quantified by employing a genetic pulse-chase technique called RITE, which was combined with chromatin immunoprecipitation and high-throughput sequencing. In this screen, the NuB4/HAT-B complex, containing the conserved type B histone acetyltransferase Hat1, was found to promote histone turnover. Unexpectedly, the three members of this complex could be functionally separated from each other as well as from the known interacting factor and histone chaperone Asf1. Thus, systematic and direct interrogation of chromatin structure on DNA barcodes can lead to the discovery of genes and pathways involved in chromatin modification and dynamics

A Rescorla-Wagner Drift-Diffusion Model of Conditioning and Timing

Computational models of classical conditioning have made significant contributions to the theoretic understanding of associative learning, yet they still struggle when the temporal aspects of conditioning are taken into account. Interval timing models have contributed a rich variety of time representations and provided accurate predictions for the timing of responses, but they usually have little to say about associative learning. In this article we present a unified model of conditioning and timing that is based on the influential Rescorla-Wagner conditioning model and the more recently developed Timing Drift-Diffusion model. We test the model by simulating 10 experimental phenomena and show that it can provide an adequate account for 8, and a partial account for the other 2. We argue that the model can account for more phenomena in the chosen set than these other similar in scope models: CSC-TD, MS-TD, Learning to Time and Modular Theory. A comparison and analysis of the mechanisms in these models is provided, with a focus on the types of time representation and associative learning rule used

Integrated log interpretation approach for underground gas storage characterization

Paper SPE 13153