Prolonged dopamine signalling in striatum signals proximity and value of distant rewards

Predictions about future rewarding events have a powerful influence on behaviour. The phasic spike activity of dopamine-containing neurons, and corresponding dopamine transients in the striatum, are thought to underlie these predictions, encoding positive and negative reward prediction errors. However, many behaviours are directed towards distant goals, for which transient signals may fail to provide sustained drive. Here we report an extended mode of reward-predictive dopamine signalling in the striatum that emerged as rats moved towards distant goals. These dopamine signals, which were detected with fast-scan cyclic voltammetry (FSCV), gradually increased or—in rare instances—decreased as the animals navigated mazes to reach remote rewards, rather than having phasic or steady tonic profiles. These dopamine increases (ramps) scaled flexibly with both the distance and size of the rewards. During learning, these dopamine signals showed spatial preferences for goals in different locations and readily changed in magnitude to reflect changing values of the distant rewards. Such prolonged dopamine signalling could provide sustained motivational drive, a control mechanism that may be important for normal behaviour and that can be impaired in a range of neurologic and neuropsychiatric disorders.National Institutes of Health (U.S.) (Grant R01 MH060379)National Parkinson Foundation (U.S.)Cure Huntington’s Disease Initiative, Inc. (Grant A-5552)Stanley H. and Sheila G. Sydney Fun

Genetic inhibition of neurotransmission reveals role of glutamatergic input to dopamine neurons in high-effort behavior

Midbrain dopamine neurons are crucial for many behavioral and cognitive functions. As the major excitatory input, glutamatergic afferents are important for control of the activity and plasticity of dopamine neurons. However, the role of glutamatergic input as a whole onto dopamine neurons remains unclear. Here we developed a mouse line in which glutamatergic inputs onto dopamine neurons are specifically impaired, and utilized this genetic model to directly test the role of glutamatergic inputs in dopamine-related functions. We found that while motor coordination and reward learning were largely unchanged, these animals showed prominent deficits in effort-related behavioral tasks. These results provide genetic evidence that glutamatergic transmission onto dopaminergic neurons underlies incentive motivation, a willingness to exert high levels of effort to obtain reinforcers, and have important implications for understanding the normal function of the midbrain dopamine system.Fil: Hutchison, M. A.. National Institutes of Health; Estados UnidosFil: Gu, X.. National Institutes of Health; Estados UnidosFil: Adrover, Martín Federico. National Institutes of Health; Estados Unidos. Consejo Nacional de Investigaciones Científicas y Técnicas. Instituto de Investigaciones en Ingeniería Genética y Biología Molecular "Dr. Héctor N. Torres"; ArgentinaFil: Lee, M. R.. National Institutes of Health; Estados UnidosFil: Hnasko, T. S.. University of California at San Diego; Estados UnidosFil: Alvarez, V. A.. National Institutes of Health; Estados UnidosFil: Lu, W.. National Institutes of Health; Estados Unido

When Does Reward Maximization Lead to Matching Law?

What kind of strategies subjects follow in various behavioral circumstances has been a central issue in decision making. In particular, which behavioral strategy, maximizing or matching, is more fundamental to animal's decision behavior has been a matter of debate. Here, we prove that any algorithm to achieve the stationary condition for maximizing the average reward should lead to matching when it ignores the dependence of the expected outcome on subject's past choices. We may term this strategy of partial reward maximization “matching strategy”. Then, this strategy is applied to the case where the subject's decision system updates the information for making a decision. Such information includes subject's past actions or sensory stimuli, and the internal storage of this information is often called “state variables”. We demonstrate that the matching strategy provides an easy way to maximize reward when combined with the exploration of the state variables that correctly represent the crucial information for reward maximization. Our results reveal for the first time how a strategy to achieve matching behavior is beneficial to reward maximization, achieving a novel insight into the relationship between maximizing and matching

Learning Priors for Bayesian Computations in the Nervous System

Our nervous system continuously combines new information from our senses with information it has acquired throughout life. Numerous studies have found that human subjects manage this by integrating their observations with their previous experience (priors) in a way that is close to the statistical optimum. However, little is known about the way the nervous system acquires or learns priors. Here we present results from experiments where the underlying distribution of target locations in an estimation task was switched, manipulating the prior subjects should use. Our experimental design allowed us to measure a subject's evolving prior while they learned. We confirm that through extensive practice subjects learn the correct prior for the task. We found that subjects can rapidly learn the mean of a new prior while the variance is learned more slowly and with a variable learning rate. In addition, we found that a Bayesian inference model could predict the time course of the observed learning while offering an intuitive explanation for the findings. The evidence suggests the nervous system continuously updates its priors to enable efficient behavior

Mathematical properties of neuronal TD-rules and differential Hebbian learning: a comparison

A confusingly wide variety of temporally asymmetric learning rules exists related to reinforcement learning and/or to spike-timing dependent plasticity, many of which look exceedingly similar, while displaying strongly different behavior. These rules often find their use in control tasks, for example in robotics and for this rigorous convergence and numerical stability is required. The goal of this article is to review these rules and compare them to provide a better overview over their different properties. Two main classes will be discussed: temporal difference (TD) rules and correlation based (differential hebbian) rules and some transition cases. In general we will focus on neuronal implementations with changeable synaptic weights and a time-continuous representation of activity. In a machine learning (non-neuronal) context, for TD-learning a solid mathematical theory has existed since several years. This can partly be transfered to a neuronal framework, too. On the other hand, only now a more complete theory has also emerged for differential Hebb rules. In general rules differ by their convergence conditions and their numerical stability, which can lead to very undesirable behavior, when wanting to apply them. For TD, convergence can be enforced with a certain output condition assuring that the δ-error drops on average to zero (output control). Correlation based rules, on the other hand, converge when one input drops to zero (input control). Temporally asymmetric learning rules treat situations where incoming stimuli follow each other in time. Thus, it is necessary to remember the first stimulus to be able to relate it to the later occurring second one. To this end different types of so-called eligibility traces are being used by these two different types of rules. This aspect leads again to different properties of TD and differential Hebbian learning as discussed here. Thus, this paper, while also presenting several novel mathematical results, is mainly meant to provide a road map through the different neuronally emulated temporal asymmetrical learning rules and their behavior to provide some guidance for possible applications

For whom is a health-promoting intervention effective? Predictive factors for performing activities of daily living independently

BACKGROUND: Health-promoting interventions tailored to support older persons to remain in their homes, so-called "ageing in place" is important for supporting or improving their health. The health-promoting programme "Elderly Persons in the Risk Zone," (EPRZ) was set up for this purpose and has shown positive results for maintaining independence in activities of daily living for older persons 80 years and above at 1- and 2 year follow-ups. The aim of this study was to explore factors for maintaining independence in the EPRZ health-promoting programme.METHODS: Total of 459 participants in the original trial was included in the analysis; 345 in the programme arm and 114 in the control arm. Thirteen variables, including demographic, health, and programme-specific indicators, were chosen as predictors for independence of activities of daily living. Logistic regression was performed separately for participants in the health promotion programme and in the control arm.RESULTS: In the programme arm, being younger, living alone and self-rated lack of tiredness in performing mobility activities predicted a positive effect of independence in activities of daily living at 1-year follow-up (odds ratio [OR] 1.18, 1.73, 3.02) and 2-year, (OR 1.13, 2.01, 2.02). In the control arm, being less frail was the only predictor at 1-year follow up (OR 1.6 1.09, 2.4); no variables predicted the outcome at the 2-year follow-up.CONCLUSIONS: Older persons living alone - as a risk of ill health - should be especially recognized and offered an opportunity to participate in health-promoting programmes such as "Elderly Persons in the Risk Zone". Further, screening for subjective frailty could form an advantageous guiding principle to target the right population when deciding to whom health-promoting intervention should be offered.TRIAL REGISTRATION: The original clinical trial was registered at ClinicalTrials.gov. Identifier: NCT00877058 , April 6, 2009

A Symbiotic Brain-Machine Interface through Value-Based Decision Making

BACKGROUND: In the development of Brain Machine Interfaces (BMIs), there is a great need to enable users to interact with changing environments during the activities of daily life. It is expected that the number and scope of the learning tasks encountered during interaction with the environment as well as the pattern of brain activity will vary over time. These conditions, in addition to neural reorganization, pose a challenge to decoding neural commands for BMIs. We have developed a new BMI framework in which a computational agent symbiotically decoded users' intended actions by utilizing both motor commands and goal information directly from the brain through a continuous Perception-Action-Reward Cycle (PARC). METHODOLOGY: The control architecture designed was based on Actor-Critic learning, which is a PARC-based reinforcement learning method. Our neurophysiology studies in rat models suggested that Nucleus Accumbens (NAcc) contained a rich representation of goal information in terms of predicting the probability of earning reward and it could be translated into an evaluative feedback for adaptation of the decoder with high precision. Simulated neural control experiments showed that the system was able to maintain high performance in decoding neural motor commands during novel tasks or in the presence of reorganization in the neural input. We then implanted a dual micro-wire array in the primary motor cortex (M1) and the NAcc of rat brain and implemented a full closed-loop system in which robot actions were decoded from the single unit activity in M1 based on an evaluative feedback that was estimated from NAcc. CONCLUSIONS: Our results suggest that adapting the BMI decoder with an evaluative feedback that is directly extracted from the brain is a possible solution to the problem of operating BMIs in changing environments with dynamic neural signals. During closed-loop control, the agent was able to solve a reaching task by capturing the action and reward interdependency in the brain

Regional movements of the tiger shark, Galeocerdo cuvier, off northeastern Brazil: inferences regarding shark attack hazard

An abnormally high shark attack rate verified off Recife could be related to migratory behavior of tiger sharks. This situation started after the construction of the Suape port to the south of Recife. A previous study suggested that attacking sharks could be following northward currents and that they were being attracted shoreward by approaching vessels. In this scenario, such northward movement pattern could imply a higher probability of sharks accessing the littoral area of Recife after leaving Suape. Pop-up satellite archival taus were deployed on five tiger sharks caught off Recife to assess their movement patterns off northeastern Brazil. All tags transmitted from northward latitudes after 7-74 days of freedom. The shorter, soak distance between deployment and pop-up locations ranged between 33-209 km and implied minimum average speeds of 0.02-0.98 km.h(-1). Both pop-up locations and depth data suggest that tiger shark movements were conducted mostly over the continental shelf. The smaller sharks moved to deeper waters within 24 hours after releasing, but they assumed a shallower (< 50 m) vertical distribution for most of the monitoring period. While presenting the first data on tiger shark movements in the South Atlantic, this study also adds new information for the reasoning of the high shark attack rate verified in this region,State Government of Pernambuco and Petrobras (Brazil); Fundacao para a Ciencia e Tecnologia (Portugal) [MCTES/FCT/SFRH/BD/37065/2007

Franck-Condon blockade in suspended carbon nanotube quantum dots

Understanding the influence of vibrational motion of the atoms on electronic transitions in molecules constitutes a cornerstone of quantum physics, as epitomized by the Franck-Condon principle of spectroscopy. Recent advances in building molecular-electronics devices and nanoelectromechanical systems open a new arena for studying the interaction between mechanical and electronic degrees of freedom in transport at the single-molecule level. The tunneling of electrons through molecules or suspended quantum dots has been shown to excite vibrational modes, or vibrons. Beyond this effect, theory predicts that strong electron-vibron coupling dramatically suppresses the current flow at low biases, a collective behaviour known as Franck-Condon blockade. Here we show measurements on quantum dots formed in suspended single-wall carbon nanotubes revealing a remarkably large electron-vibron coupling and, due to the high quality and unprecedented tunability of our samples, admit a quantitative analysis of vibron-mediated electronic transport in the regime of strong electron-vibron coupling. This allows us to unambiguously demonstrate the Franck-Condon blockade in a suspended nanostructure. The large observed electron-vibron coupling could ultimately be a key ingredient for the detection of quantized mechanical motion. It also emphasizes the unique potential for nanoelectromechanical device applications based on suspended graphene sheets and carbon nanotubes.Comment: 7 pages, 3 figure