The Psychophysics of Reward: Empirical Studies and Modeling of Performance for Medial Forebrain Electrical Stimulation in the Rat

Brain stimulation reward (BSR), the effect of electrical stimulation that animals seek to reinitiate, is a useful tool to investigate reward-seeking behaviour and its neural underpinnings. The experiments in this thesis pursue this approach by applying the "reward-mountain" model of performance for BSR. This model provides a framework for describing the computational processes that link the induced neural activity to reward- seeking behaviour. The data to which the model is fit are obtained by measuring operant performance for BSR (time spent pressing a lever) as a function of subjective intensity of the stimulation (controlled by pulse frequency) and opportunity cost (work time required to earn a reward). Determining the stage of neural circuitry responsible for the behavioural impact of physiological manipulations is among the principal goals of this strategy. At the core of the model is the subject’s computation of “payoff” via the integration of reward intensity and costs. An important initial stage, often overlooked in neuroscientific studies of decision-making, is the transformation of the objective into subjective variables. The formal relationship between these variables (termed psychophysical functions) is often non-linear: what is experienced is not necessarily a direct reflection of the external world. An analysis of these transformations is important for the full understanding of cost-benefit decision-making. A central goal of the experiments in this thesis is to estimate the psychophysical functions of reward-seeking variables. Chapter 1 reviews the BSR literature and describes the reward-mountain model. The experiment described in Chapter 2 concerns the valuation of time: the translation of the experimenter-set opportunity cost (the objective price) into the equivalent subjective domain (subjective price). The experiment described in Chapter 3 estimates the frequency-response function of the directly stimulated neurons subserving the rewarding effect. This function translates the experimenter-set pulse frequency (the inducing stimulus) into the firing frequency of the neuron (the induced physiological response). Chapter 4 describes a proof-of-principle study: the ability of the reward-mountain paradigm to detect the effect of a lesion challenge on pursuit of BSR and to link this effect to one or more stages of processing. Chapter 5 concludes with a general discussion

Subjective estimates of opportunity cost in rats working for rewarding brain stimulation

The principal goal of psychophysics is to describe the functions that transform the objective variables into their subjective equivalents. The matching law has been used to describe the function translating the objective strength of a rewarding stimulus (e.g., the concentration of a sucrose solution) into its subjective impact. In contrast, the psychophysics of the cost of a reward has been little studied. This is a salient lacuna. The present experiments examine opportunity cost, the price paid for a reward in terms of the time taken away from competing activities, such as procurement of alternate rewards, resting, grooming, and exploration. The reward is electrical stimulation delivered to the medial forebrain bundle. It is the subjective interpretation of these values (reward strength and cost) that an animal uses for goal selection. A widely held assumption is that the animal computes the payoff of a goal (an index of how worthwhile it is for it to choose a goal) as the ratio of the subjective reward strength to the subjective cost required to obtain the goal. This assumption is the basis for the present experiments: changes in subjective reward strength (manipulated by the frequency of stimulation) are used to compensate for changes in opportunity costs by the animal, and changes in opportunity costs are used to compensate for changes in reward strength. The present experiments estimate the function that transforms objective opportunity costs into subjective opportunity costs under the assumed definition of payoff. In Experiment I, the change in frequency required to offset a constant proportional change in price is determined, providing an estimate of the first derivative of the subjective-price function. This experiment demonstrates that as the time intervals are shortened, subjective opportunity cost levels off and deviates substantially from the objective cost. In Experiment II, the change in price required to offset a fixed difference in frequency is determined which provides an estimate of the subjective-price function itself. This experiment demonstrates that subjective costs approximate objective ones when the time intervals involved are relatively long. The two experiments complemented each other: the first revealed the scalar range of the objective-subjective cost relationship while the second demonstrated where this relationship breaks down. It has been implicitly assumed that the relation between objective and subjective costs approximate each other; these experiments showed that this is true but that the relationship breaks down at low costs. The function mapping objective costs into subjective ones (the "subjective-cost function") would prove useful because if the cost of a reward were to be used as a parameter that is manipulated in other experiments, it is important to choose a value of cost that the animal will accurately interpre

Optimal indolence: a normative microscopic approach to work and leisure

Dividing limited time between work and leisure when both have their attractions is a common everyday decision. We provide a normative control theoretic treatment of this decision that bridges economic and psychological accounts. We show how our framework applies to free-operant behavioural experiments in which subjects are required to work (depressing a lever) for sufficient total time (called the price) to receive a reward. When the microscopic benefit-of-leisure increases nonlinearly with duration, the model generates behaviour that qualitatively matches various microfeatures of subjects’ choices, including the distribution of leisure bout durations as a function of the payoff. We relate our model to traditional accounts by deriving macroscopic, molar, quantities from microscopic choices

Valuation of opportunity costs by rats working for rewarding electrical brain stimulation - Fig 1

<p><b>A: Objective-price function.</b> Subjective prices are equal to objective prices, as assumed in early studies using the reward-mountain model [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0182120#pone.0182120.ref014" target="_blank">14</a>–<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0182120#pone.0182120.ref017" target="_blank">17</a>]. <b>B: Sigmoidal-slope function.</b> The functional form resembles a hockey stick. The flat “blade” of the function denotes the range over which costs are subjectively invariant. That portion merges with an upward-bending segment over which the costs are discriminable, but do not yet rise at the same rate as the objective price. The final “handle” portion denotes the range over which the relationship between objective and subjective costs is scalar. Changing the value of the parameter shifts the curve vertically, whereas changing the value of the parameter alters the abruptness of the transition between the blade and the handle. <b>C: Linear-price function.</b> This is the subjective-price function derived from Mazur’s [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0182120#pone.0182120.ref032" target="_blank">32</a>] hyperbolic-discounting equation. The greater the value of <i>K</i>_<i>h</i>, the faster the function rises. In order for the terminal slope to approach unity, <i>K</i>_<i>h</i> must equal 1. (The lines look curved because they are plotted in double logarithmic coordinates; in linear coordinates, they would be straight.) <b>D: Exponential-price function.</b> This is the subjective-price function derived from exponential discounting. The larger the <i>K</i>_<i>x</i> value, the more rapidly the function rises. See <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0182120#pone.0182120.s001" target="_blank">S1 File</a> for a discussion of units.</p

The contours of the reward mountain reflect the form of the reward-intensity-growth function.

<p>Upper panel: the reward-intensity-growth function, as described by Simmons and Gallistel [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0182120#pone.0182120.ref013" target="_blank">13</a>] and by Sonnenschein, Conover and Shizgal [<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0182120#pone.0182120.ref046" target="_blank">46</a>]; the parameters {<i>F</i>_<i>hm</i>,<i>g</i>} are from the fit of the objective-price function to the data from rat F18. Middle panel: Contour map of the reward-mountain variant that incorporates the objective-price function; the parameters {<i>a</i>,<i>F</i>_<i>hm</i>,<i>g</i>} are again from the fit of the objective-price function to the data from rat F18. The contours are rotated traces of the reward-intensity-growth function in the upper panel. Lower panel: Contour map of the reward-mountain variant that incorporates the sigmoidal-slope function; the parameters {<i>a</i>,<i>F</i>_<i>hm</i>,<i>g</i>,,} are from the fit of the sigmoidal-slope function to the data from rat F18. These contours reflect the non-linear form of both the reward-intensity-growth and subjective-price functions. The contours defined by the reward-mountain variants incorporating the objective- and linear-price functions (not shown) also bend toward the horizontal at low prices.</p

Sigmoidal-slope function: Estimated parameter values.

<p>Sigmoidal-slope function: Estimated parameter values.</p

The reward mountain.

<p>Allocation of time to reward seeking as a function of the cost and strength of reward. <b>A</b>: The surface predicted by the version of the reward-mountain model incorporating the objective-price function. <b>B</b>: The surface predicted by the version of the reward-mountain model incorporating the sigmoidal-slope function. The heavy black line traces the contour corresponding to “mid-range” time allocation (half-way between the minimal and maximal values). Given that the two surfaces differ in shape, we can choose between the subjective-price functions embedded in the respective models (and in the linear-price and exponential-price functions as well) by determining which surface best fits the data. Note that the x-axes represent <i>P</i>_<i>obj</i>, the objective price. The subjective-price functions determine how <i>P</i>_<i>obj</i> is translated into <i>P</i>_<i>sub</i>, the subjective price of the reward. The surfaces produced by the version of the reward-mountain model incorporating the linear- and exponential-price models can be seen in <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0182120#pone.0182120.g005" target="_blank">Fig 5<b>C</b> and 5<b>D</b></a>.</p

Time allocation as a function of the strength and cost of reward.

<p>The colored symbols represent the proportion of trial time allocated to reward seeking by rat F16 as a function of price and pulse frequency. The corresponding legend and contour plots are presented in <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0182120#pone.0182120.g006" target="_blank">Fig 6</a>. Each of the fitted surfaces is defined by one of the four subjective-price functions (Eqs <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0182120#pone.0182120.e013" target="_blank">7</a>, <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0182120#pone.0182120.e018" target="_blank">9</a>, <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0182120#pone.0182120.e024" target="_blank">12</a> and <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0182120#pone.0182120.e025" target="_blank">13</a>). Analogous plots for the remaining rats are shown in Figs A, B, C, E and F in <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0182120#pone.0182120.s001" target="_blank">S1 File</a>.</p

Comparison between interpolated data points and pulse-frequency-versus-objective-price trade-off functions derived from the surface fits.

<p>The solid line is the contour in <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0182120#pone.0182120.g006" target="_blank">Fig 6</a> representing mid-range time allocation (half-way between <i>T</i>_<i>min</i> and <i>T</i>_<i>max</i>) by rat F16. The corresponding data points were interpolated by means of spline fits to the data from the pulse-frequency, price, and radial pseudo-sweeps. Analogous plots for the remaining rats are shown in Figs M, N, O, Q and R in <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0182120#pone.0182120.s001" target="_blank">S1 File</a>.</p

Model comparison.

<p>Model comparison.</p