A neural network model of adaptively timed reinforcement learning and hippocampal dynamics

A neural model is described of how adaptively timed reinforcement learning occurs. The adaptive timing circuit is suggested to exist in the hippocampus, and to involve convergence of dentate granule cells on CA3 pyramidal cells, and NMDA receptors. This circuit forms part of a model neural system for the coordinated control of recognition learning, reinforcement learning, and motor learning, whose properties clarify how an animal can learn to acquire a delayed reward. Behavioral and neural data are summarized in support of each processing stage of the system. The relevant anatomical sites are in thalamus, neocortex, hippocampus, hypothalamus, amygdala, and cerebellum. Cerebellar influences on motor learning are distinguished from hippocampal influences on adaptive timing of reinforcement learning. The model simulates how damage to the hippocampal formation disrupts adaptive timing, eliminates attentional blocking, and causes symptoms of medial temporal amnesia. It suggests how normal acquisition of subcortical emotional conditioning can occur after cortical ablation, even though extinction of emotional conditioning is retarded by cortical ablation. The model simulates how increasing the duration of an unconditioned stimulus increases the amplitude of emotional conditioning, but does not change adaptive timing; and how an increase in the intensity of a conditioned stimulus "speeds up the clock", but an increase in the intensity of an unconditioned stimulus does not. Computer simulations of the model fit parametric conditioning data, including a Weber law property and an inverted U property. Both primary and secondary adaptively timed conditioning are simulated, as are data concerning conditioning using multiple interstimulus intervals (ISIs), gradually or abruptly changing ISis, partial reinforcement, and multiple stimuli that lead to time-averaging of responses. Neurobiologically testable predictions are made to facilitate further tests of the model.Air Force Office of Scientific Research (90-0175, 90-0128); Defense Advanced Research Projects Agency (90-0083); National Science Foundation (IRI-87-16960); Office of Naval Research (N00014-91-J-4100

The hippocampus and cerebellum in adaptively timed learning, recognition, and movement

The concepts of declarative memory and procedural memory have been used to distinguish two basic types of learning. A neural network model suggests how such memory processes work together as recognition learning, reinforcement learning, and sensory-motor learning take place during adaptive behaviors. To coordinate these processes, the hippocampal formation and cerebellum each contain circuits that learn to adaptively time their outputs. Within the model, hippocampal timing helps to maintain attention on motivationally salient goal objects during variable task-related delays, and cerebellar timing controls the release of conditioned responses. This property is part of the model's description of how cognitive-emotional interactions focus attention on motivationally valued cues, and how this process breaks down due to hippocampal ablation. The model suggests that the hippocampal mechanisms that help to rapidly draw attention to salient cues could prematurely release motor commands were not the release of these commands adaptively timed by the cerebellum. The model hippocampal system modulates cortical recognition learning without actually encoding the representational information that the cortex encodes. These properties avoid the difficulties faced by several models that propose a direct hippocampal role in recognition learning. Learning within the model hippocampal system controls adaptive timing and spatial orientation. Model properties hereby clarify how hippocampal ablations cause amnesic symptoms and difficulties with tasks which combine task delays, novelty detection, and attention towards goal objects amid distractions. When these model recognition, reinforcement, sensory-motor, and timing processes work together, they suggest how the brain can accomplish conditioning of multiple sensory events to delayed rewards, as during serial compound conditioning.Air Force Office of Scientific Research (F49620-92-J-0225, F49620-86-C-0037, 90-0128); Advanced Research Projects Agency (ONR N00014-92-J-4015); Office of Naval Research (N00014-91-J-4100, N00014-92-J-1309, N00014-92-J-1904); National Institute of Mental Health (MH-42900

Cerebellar Learning in an Opponent Motor Controller for Adaptive Load Compensation and Synergy Formation

This paper shows how a minimal neural network model of the cerebellum may be embedded within a sensory-neuro-muscular control system that mimics known anatomy and physiology. With this embedding, cerebellar learning promotes load compensation while also allowing both coactivation and reciprocal inhibition of sets of antagonist muscles. In particular, we show how synaptic long term depression guided by feedback from muscle stretch receptors can lead to trans-cerebellar gain changes that are load-compensating. It is argued that the same processes help to adaptively discover multi-joint synergies. Simulations of rapid single joint rotations under load illustrates design feasibility and stability.National Science Foundation (IRI-90-24877, IRI-87-16960); Office of Naval Research (N00014-92-J-1309); Consejo Nacional de Ciencia y Technología (63462); Air Force Office of Scientific Research (F49620-92-J-0499); Defense Advanced Research Projects Agency (AFOSR 90-0083, ONR N00014-92-J-4015

Equilibria and Dynamics of a Neural Network Model for Opponent Muscle Control

One of the advantages of biological skeleto-motor systems is the opponent muscle design, which in principle makes it possible to achieve facile independent control of joint angle and joint stiffness. Prior analysis of equilibrium states of a biologically-based neural network for opponent muscle control, the FLETE model, revealed that such independent control requires specialized interneuronal circuitry to efficiently coordinate the opponent force generators. In this chapter, we refine the FLETE circuit variables specification and update the equilibrium analysis. We also incorporate additional neuronal circuitry that ensures efficient opponent force generation and velocity regulation during movement.National Science Foundation (IRI-90-24877); Consejo Nacional de Ciencia y Tecnologia, Méxic

A Spectral Network Model of Pitch Perception

A model of pitch perception, called the Spatial Pitch Network or SPINET model, is developed and analyzed. The model neurally instantiates ideas front the spectral pitch modeling literature and joins them to basic neural network signal processing designs to simulate a broader range of perceptual pitch data than previous spectral models. The components of the model arc interpreted as peripheral mechanical and neural processing stages, which arc capable of being incorporated into a larger network architecture for separating multiple sound sources in the environment. The core of the new model transforms a spectral representation of an acoustic source into a spatial distribution of pitch strengths. The SPINET model uses a weighted "harmonic sieve" whereby the strength of activation of a given pitch depends upon a weighted sum of narrow regions around the harmonics of the nominal pitch value, and higher harmonics contribute less to a pitch than lower ones. Suitably chosen harmonic weighting functions enable computer simulations of pitch perception data involving mistuned components, shifted harmonics, and various types of continuous spectra including rippled noise. It is shown how the weighting functions produce the dominance region, how they lead to octave shifts of pitch in response to ambiguous stimuli, and how they lead to a pitch region in response to the octave-spaced Shepard tone complexes and Deutsch tritones without the use of attentional mechanisms to limit pitch choices. An on-center off-surround network in the model helps to produce noise suppression, partial masking and edge pitch. Finally, it is shown how peripheral filtering and short term energy measurements produce a model pitch estimate that is sensitive to certain component phase relationships.Air Force Office of Scientific Research (F49620-92-J-0225); American Society for Engineering Educatio

Recognition of 3-D Objects from Multiple 2-D Views by a Self-Organizing Neural Architecture

The recognition of 3-D objects from sequences of their 2-D views is modeled by a neural architecture, called VIEWNET that uses View Information Encoded With NETworks. VIEWNET illustrates how several types of noise and varialbility in image data can be progressively removed while incornplcte image features are restored and invariant features are discovered using an appropriately designed cascade of processing stages. VIEWNET first processes 2-D views of 3-D objects using the CORT-X 2 filter, which discounts the illuminant, regularizes and completes figural boundaries, and removes noise from the images. Boundary regularization and cornpletion are achieved by the same mechanisms that suppress image noise. A log-polar transform is taken with respect to the centroid of the resulting figure and then re-centered to achieve 2-D scale and rotation invariance. The invariant images are coarse coded to further reduce noise, reduce foreshortening effects, and increase generalization. These compressed codes are input into a supervised learning system based on the fuzzy ARTMAP algorithm. Recognition categories of 2-D views are learned before evidence from sequences of 2-D view categories is accumulated to improve object recognition. Recognition is studied with noisy and clean images using slow and fast learning. VIEWNET is demonstrated on an MIT Lincoln Laboratory database of 2-D views of jet aircraft with and without additive noise. A recognition rate of 90% is achieved with one 2-D view category and of 98.5% correct with three 2-D view categories.National Science Foundation (IRI 90-24877); Office of Naval Research (N00014-91-J-1309, N00014-91-J-4100, N00014-92-J-0499); Air Force Office of Scientific Research (F9620-92-J-0499, 90-0083

Responder analysis of a randomized comparison of the 13.3 mg/24 h and 9.5 mg/24 h rivastigmine patch

INTRODUCTION: OPtimizing Transdermal Exelon In Mild-to-moderate Alzheimer’s disease (OPTIMA) was a randomized, double-blind comparison of 13.3 mg/24 h versus 9.5 mg/24 h rivastigmine patch in patients with mild-to-moderate Alzheimer’s disease who declined despite open-label treatment with 9.5 mg/24 h patch. Over 48 weeks of double-blind treatment, high-dose patch produced greater functional and cognitive benefits compared with 9.5 mg/24 h patch. METHODS: Using OPTIMA data, a post-hoc responder analysis was performed to firstly, compare the proportion of patients demonstrating improvement or absence of decline with 13.3 mg/24 h versus 9.5 mg/24 h patch; and secondly, identify predictors of improvement or absence of decline. ‘Improvers’ were patients who improved on the Alzheimer’s Disease Assessment Scale–cognitive subscale (ADAS-cog) by ≥4 points from baseline, and did not decline on the instrumental domain of the Alzheimer’s Disease Cooperative Study–Activities of Daily Living scale (ADCS-IADL). ‘Non-decliners’ were patients who did not decline on either scale. RESULTS: Overall, 265 patients randomized to 13.3 mg/24 h and 271 to 9.5 mg/24 h patch met the criteria for inclusion in the intention-to-treat population and were included in the analyses. Significantly more patients were ‘improvers’ with 13.3 mg/24 h compared with 9.5 mg/24 h patch at Weeks 24 (44 (16.6%) versus 19 (7.0%); P < 0.001) and 48 (21 (7.9%) versus 10 (3.7%); P = 0.023). A significantly greater proportion of patients were ‘non-decliners’ with 13.3 mg/24 h compared with 9.5 mg/24 h patch at Week 24 (71 (26.8%) versus 44 (16.2%); P = 0.002). At Week 48, there was a trend in favor of 13.3 mg/24 h patch. Functional and cognitive assessment scores at double-blind baseline did not consistently predict effects at Weeks 24 or 48. CONCLUSION: More patients with mild-to-moderate Alzheimer’s disease who are titrated to 13.3 mg/24 h rivastigmine patch at time of decline are ‘improvers’ or ‘non-decliners’ i.e. show responses on cognition and activities of daily living compared with patients remaining on 9.5 mg/24 h patch. TRIAL REGISTRATION: Clinicaltrials.gov identifier: NCT00506415; registered July 20, 2007. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (doi:10.1186/s13195-014-0088-8) contains supplementary material, which is available to authorized users

Trial Designs Likely to Meet Valid Long-Term Alzheimer's Disease Progression Effects: Learning from the Past, Preparing for the Future

The International Society for CNS Clinical Trials and Methodology (ISCTM) held its 4th Annual Autumn Conference in Toronto, Ontario, October 6-7, 2008. The purpose of the present report is to provide an overview of one of the sessions at the conference which focused on the designs and methodologies to be applied in clinical trials of new treatments for Alzheimer's disease (AD) with purported “disease-modifying” effects. The session began with a discussion of how neuroimaging has been applied in multiple sclerosis clinical trials (another condition for which disease modification claims have been achieved). The next two lectures provided a pharmaceutical industry perspective on some of the specific challenges and possible solutions for designing trials to measure disease progression and/or modification. The final lecture provided an academic viewpoint and the closing discussion included additional academic and regulatory perspectives on trial designs, methodologies, and statistical issues relevant to the disease modification concept

A Stable Biologically Motivated Learning Mechanism for Visual Feature Extraction to Handle Facial Categorization

The brain mechanism of extracting visual features for recognizing various objects has consistently been a controversial issue in computational models of object recognition. To extract visual features, we introduce a new, biologically motivated model for facial categorization, which is an extension of the Hubel and Wiesel simple-to-complex cell hierarchy. To address the synaptic stability versus plasticity dilemma, we apply the Adaptive Resonance Theory (ART) for extracting informative intermediate level visual features during the learning process, which also makes this model stable against the destruction of previously learned information while learning new information. Such a mechanism has been suggested to be embedded within known laminar microcircuits of the cerebral cortex. To reveal the strength of the proposed visual feature learning mechanism, we show that when we use this mechanism in the training process of a well-known biologically motivated object recognition model (the HMAX model), it performs better than the HMAX model in face/non-face classification tasks. Furthermore, we demonstrate that our proposed mechanism is capable of following similar trends in performance as humans in a psychophysical experiment using a face versus non-face rapid categorization task

Efficacy of Higher Dose 13.3 mg/24 h Rivastigmine Patch on Instrumental Activities of Daily Living in Patients with Mild-to-Moderate Alzheimer's Disease

Background: Stabilizing/reducing decline in the ability to perform activities of daily living (ADLs) is important in management of Alzheimer's disease (AD). Methods: Post hoc analysis of OPtimizing Transdermal Exelon In Mild-to-moderate Alzheimer's disease (OPTIMA), a double-blind trial comparing 13.3 and 9.5 mg/24 h rivastigmine patch in patients with AD demonstrating functional and cognitive decline with 9.5 mg/24 h patch. Efficacy on Alzheimer's disease Cooperative Study-instrumental ADL (ADCS-IADL) items, higher level function (HLF), and autonomy factors was assessed. Results: The ADCS-IADL, HLF, and autonomy factors favored 13.3 mg/24 h patch at all time points, reaching significance from weeks 16 to 48, 24 to 48, and 32 to 48, respectively. Higher dose patch demonstrated significantly greater efficacy on 10 of 17 ADCS-IADL items at 1 or more time points ( P < .05 vs 9.5 mg/24 h patch). More adverse events were observed with higher dose patch; study discontinuations were similar between the doses. Conclusions: Greater efficacy of 13.3 versus 9.5 mg/24 h patch on ADL, including autonomy and HLF factors, supports this additional dosing option to prolong patients' independence