On the neural computation of utility

The rewarding effect produced by electrical stimulation of the lateral hypothalamus can compete and summate with gustatory rewards. However, physiological manipulations, such as sodium depletion and the accumulation of an energy-rich solution in the gut, can alter the rewarding impact of the gustatory stimuli without producing substantial changes in the rewarding effect of the electrical stimulation. On the basis of their competition and summation, it is argued that the artificial and natural rewards are evaluated in a common currency, represented in an aggregate firing-rate code. Such a code would make it possible for the synchronous, spatially contiguous pattern of neural firing induced by the electrode to simulate a signal normally produced by asynchronous, spatially distributed activity. It is suggested that a unidimensional code of this sort is employed to represent the utility of a goal object. In order for physiological feedback to alter the utility of one natural reward, such as sucrose, without changing the utility of a second natural reward, such as a salt solution, the physiological feedback signals must enter into the computation of utility at a stage of processing in which the representations of the two natural rewards are distinct. However, orderly choice between such rewards implies that their utilities are expressed ultimately in a common neural currency. That physiological feedback alters the rewarding effects of the gustatory stimuli suggests that the physiological feedback signals modulate the value of such natural stimuli at a stage of processing prior to their translation into a common currency. In contrast, physiological feedback would fail to alter the rewarding effect of the electrical stimulation if the electrically evoked signal is injected at a later stage processing, a stage in which different rewards are represented in a common currency. In this view, the signal injected by the electrical stimulation mimics the utility of a natural stimulus but not its sensory quality

Scarce Means with Alternative Uses: Robbins’ Definition of Economics and Its Extension to the Behavioral and Neurobiological Study of Animal Decision Making

Almost 80 years ago, Lionel Robbins proposed a highly influential definition of the subject matter of economics: the allocation of scarce means that have alternative ends. Robbins confined his definition to human behavior, and he strove to separate economics from the natural sciences in general and from psychology in particular. Nonetheless, I extend his definition to the behavior of non-human animals, rooting my account in psychological processes and their neural underpinnings. Some historical developments are reviewed that render such a view more plausible today than would have been the case in Robbins’ time. To illustrate a neuroeconomic perspective on decision making in non-human animals, I discuss research on the rewarding effect of electrical brain stimulation. Central to this discussion is an empirically based, functional/computational model of how the subjective intensity of the electrical reward is computed and combined with subjective costs so as to determine the allocation of time to the pursuit of reward. Some successes achieved by applying the model are discussed, along with limitations, and evidence is presented regarding the roles played by several different neural populations in processes posited by the model. I present a rationale for marshaling convergent experimental methods to ground psychological and computational processes in the activity of identified neural populations, and I discuss the strengths, weaknesses, and complementarity of the individual approaches. I then sketch some recent developments that hold great promise for advancing our understanding of structure–function relationships in neuroscience in general and in the neuroeconomic study of decision making in particular

On the neural computation of utility: implications from studies of brain stimulation reward

1. Like other vertebrates, from goldfish to humans, rats will work in order to deliver electrical stimulation to certain brain sites. Although the stimulation produces no evident physiological benefit, it is sought out avidly, as if it were a biologically significant resource. Thus, it has long been thought that the rewarding stimulation activates neural circuitry involved in the evaluation and selection of goals. 2. Computing the utility of goal objects involves a tightly integrated set of perceptual, cognitive, and motivational mechanisms. I argue that rewarding electrical brain stimulation engages only a subset of these mechanisms. If so, comparison of the ways in which the utility of electrical brain stimulation and natural reinforcers are computed may highlight operating principles and isolate components of the computational mechanisms. 3. In the view proposed here, information about goal objects and consummatory acts is processed, in parallel, in three different channels. 3.1. Perceptual processing indicates what and where the goal object is. 3.2. A stopwatch-like interval timer predicts when or how often the goal object will be available. 3.3. Under the influence of information about the current physiological state, an evaluative channel returns a subjective weighting of strength variables such as the concentration of a sucrose solution or the temperature of an air current. 3.4. The output of these channels is recorded in multidimensional records that include 3.4.1. information of perceptual origin about amount and kind (e.g., food, water,or salt) 3.4.2. information from the timer about rate and delay 3.4.3. a subjective assessment of intensity provided by the evaluative channel 4. This chapter addresses the relationships between brain stimulation reward (BSR), the perceptual, interval timing, and evaluative channels, and the variants of utility proposed by Kahneman and his coworkers on the basis of their studies of evaluation and choice in human subjects. 4.1. It is argued that the output of the evaluative channel can be manifested in experience as pleasure or suffering but that awareness is not necessary in order for this signal to influence action. 5. The neural signal injected by rewarding electrical stimulation is portrayed as providing meaningful information about rate, delay and intensity but not about amount or kind. This proposal is used to account for 5.1. competition and summation between BSR and natural rewards 5.2. differential effects of physiological feedback on the utility of BSR and natural rewards 5.3. matching of behavioral allocation to the relative rates and intensities of BSR 5.4. differences in the elasticity of demand for BSR and food in a closed economy 5.5. the high substitutability of BSR for food and water in an open economy 6. The powerful aftereffect of BSR that potentiates efforts to obtain additional stimulation is related to expectancy

Neural Basis of Utility Estimation

The allocation of behavior among competing activities and goal objects depends on the payoffs they provide.Payoff is evaluated among multiple dimensions including intensity, rate, delay, and kind. Recent findings suggest that by triggering a stream of action potentials in myelinated, medial forebrain bundle axons, rewarding electrical brain stimulation delivers a meaningful intensity signal to the process that computes payoff

Brain Stimulation Reward

In 1953, Olds and Milner discovered that rats would readily learn to work for electrical stimulation of certain brain sites. Their findings inspired a large body of research on the neural basis of reward, motivation, and learning. Unlike consummatory behaviors, which satiate as a result of ingestion of and contact with the goal object, performance for rewarding brain stimulation is remarkably stable and persistent. Pursuit of the stimulation is enhanced by different classes of dependence-inducing drugs, suggesting that common neural mechanisms underlie the rewarding effects of drugs and electrical brain stimulation. Indeed, dopamine-containing neurons in the midbrain are implicated in both phenomena. Major schools of thought that have addressed brain stimulation reward differ with regards to the roles played by hedonic experience and craving, although there is substantial overlap between the different viewpoints. A tradition that arose in the study of machine learning has been brought to bear on the role of dopamine neurons in reward-related learning in animals and on the phenomenon of intracranial self-stimulation. Neuroeconomic perspectives strive to integrate the processing of benefits, costs, and risks into an account of decision making grounded in brain circuitry. Adjudication of the differences between the various viewpoints and progress towards identifying the relevant neural circuitry has been hindered by the lack of specificity inherent in the use of electrical stimulation to study central nervous system function. Many neurons in addition to those targeted are likely activated by such stimulation. Recently developed optogenetic methods may overcome this obstacle, providing much more specific means for stimulating or silencing populations of nerve cells selected on the basis of their gene expression, cell-body location, and projections. Coupled with behavioral methods of increasing sophistication and specificity and with quantitative modeling of signal flow in the relevant neural circuitry, the new optogenetic methods promise to bring us much closer to fulfillment of the hopes engendered long ago by the discovery of brain stimulation reward

Video illustrating the reward-mountain model

This video reveals a fundamental source of ambiguity in two-dimensional measurements of operant performance for reward, such as those obtained in the curve-shift and progressive-ratio paradigms. We show how the three-dimensional portrayal provided by the reward-mountain model resolves this ambiguity

A new view of the effect of dopamine receptor antagonism on operant performance for rewarding brain stimulation in the rat

RATIONALE:Previous studies of neuroleptic challenges to intracranial self-stimulation (ICSS) employed two-dimensional (2D) measurements (curve shifts). Results so obtained are ambiguous with regard to the stage of neural processing at which the drug produces its performance-altering effect. We substituted a three-dimensional (3D) method that measures reward-seeking as a function of both the strength and cost of reward. This method reveals whether changes in reward seeking are due to drug action prior to the output of the circuitry that performs spatiotemporal integration of the stimulation-induced neural activity. OBJECTIVES:The aim of this study was to obtain new information about the stage of neural processing at which pimozide acts to alter pursuit of brain stimulation reward (BSR). METHODS:Following treatment with pimozide (0.1 mg/kg) or its vehicle, the proportion of trial time allocated to working for BSR was measured as a function of pulse frequency and opportunity cost. A surface defined by Shizgal's reward-mountain model was fitted to the drug and vehicle data. RESULTS:Pimozide lowered the cost required to decrease performance for a maximal BSR to half its maximal level but did not alter the pulse-frequency required to produce a reward of half-maximal intensity. CONCLUSIONS:Like indirect dopamine agonists, pimozide does not alter the sensitivity of brain reward circuity but changes reward-syste

Rattus Psychologicus: Construction of preferences by self-stimulating rats

Behavioral economists have proposed that human preferences are constructed during their elicitation and are thus influenced by the elicitation procedure. For example, different preferences are expressed when options are encountered one at a time or concurrently. This phenomenon has been attributed to differences in the “evaluability” of a particular attribute when comparison to an option with a different value of this attribute is or is not available. Research on the preferences of laboratory animals has often been carried out by means of operant conditioning methods. Formal treatments of operant behavior relate preferences to variables such as the strength and cost of reward but do not address the evaluability of these variables. Two experiments assessed the impact of procedural factors likely to alter the evaluability of an opportunity cost (“price”): the work time required for a rat to earn a train of rewarding electrical brain stimulation. The results support the notion that comparison between recently encountered prices is necessary to render the price variable highly evaluable. When price is held constant over many trials and test sessions, the evaluability of this variable appears to decline. Implications are discussed for the design of procedures for estimating subjective reward strengths and costs in operant conditioning experiments aimed at characterizing, identifying and understanding neural circuitry underlying evaluation and choice