Elaboration of a model of Pavlovian learning and performance: HeiDI

The model elaborated here adapts the influential pooled error term, first described by Allan R. Wagner and his colleague Robert A. Rescorla, to govern the formation of reciprocal associations between any pair of stimuli that are presented on a given trial. In the context of Pavlovian conditioning, these stimuli include various conditioned and unconditioned stimuli. This elaboration enables the model to deal with cue competition phenomena, including the relative validity effect, and evidence implicating separate error terms and attentional processes in association formation. The model also includes a performance rule, which provides a natural basis for (individual) variation in the strength and nature of conditioned behaviors that are observed in Pavlovian conditioning procedures. The new model thereby begins to address theoretical and empirical issues that were apparent when the Rescorla-Wagner model was first described, together with research inspired by the model over ensuing 50 years

Individual differences in the nature of conditioned behavior across a conditioned stimulus: adaptation and application of a model

Pavlovian conditioning procedures produce marked individual differences in the form of conditioned behavior. For example, when rats are given conditioning trials in which the temporary insertion of a lever into an operant chamber (the conditioned stimulus, CS) is paired with the delivery of food (the unconditioned stimulus, US), they exhibit knowledge of the leverfood relationship in different ways. For some rats (known as sign-trackers) interactions with the lever dominate, while for others (goal-trackers) approaching the food well dominates. A formal model of Pavlovian conditioning (HeiDI) attributes such individual differences in behavior to variations in the perceived salience of the CS and US. An application of the model in which the perceived salience of the CS declines (i.e., adapts) across its duration, predicts changes in these individual differences within the presentation of the CS: The sign-tracking bias is predicted to decline and goal-tracking bias is predicted to increase across the presentation of a lever. The accuracy of these predictions was confirmed though analysis of archival data from female and male rats

Conditioning with spatio-temporal patterns: Constraining the contribution of the hippocampus to configural learning

The conditions under which the hippocampus contributes to learning about spatio-temporal configural patterns are not fully established. The aim of Experiments 1–4 was to investigate the impact of hippocampal lesions on learning about where or when a reinforcer would be delivered. In each experiment, the rats received exposure to an identical set of patterns (i.e., spotted + morning, checked + morning, spotted + afternoon and checked + afternoon); and the contexts (Experiment 1), times of day (Experiment 2), or their configuration (Experiments 3 and 4) signalled whether or not a reinforcer would be delivered. The fact that hippocampal damage did not disrupt the formation of simple or configural associations involving spatio-temporal patterns is surprising, and suggests that the contribution of the hippocampus is restricted to mediated learning (or updating) involving spatio-temporal configurations

Higher-order conditioning: What is learnt and how it is expressed

Pairing a neutral conditioned stimulus (CS) with a motivationally significant unconditioned stimulus (US) results in the CS coming to elicit conditioned responses (CRs). The widespread significance and translational value of Pavlovian conditioning are increased by the fact that pairing two neutral CSs (A and X) enables conditioning with X to affect behavior to A. There are two traditional informal accounts of such higher-order conditioning, which build on more formal associative analyses of Pavlovian conditioning. But, higher-order conditioning and Pavlovian conditioning have characteristics that are beyond these accounts: Notably, the two are influenced in different ways by the same experimental manipulations, and both generate conditioned responses that do not reflect the US per se. Here, we present a formal analysis that sought to address these characteristics

Variation in the effectiveness of reinforcement and nonreinforcement in generating different conditioned behaviors

Rat autoshaping procedures generate two readily measurable conditioned responses: During lever presentations that have previously signaled food, rats approach the food well (called goal-tracking) and interact with the lever itself (called sign-tracking). We investigated how reinforced and nonreinforced trials affect the overall and temporal distributions of these two responses across 10-second lever presentations. In two experiments, reinforced trials generated more goal-tracking than sign-tracking, and nonreinforced trials resulted in a larger reduction in goal-tracking than sign-tracking. The effect of reinforced trials was evident as an increase in goal-tracking and reduction in sign-tracking across the duration of the lever presentations, and nonreinforced trials resulted in this pattern transiently reversing and then becoming less evident with further training. These dissociations are consistent with a recent elaboration of the Rescorla-Wagner model, HeiDI (Honey, R.C., Dwyer, D.M., & Iliescu, A.F. (2020b). HeiDI: A model for Pavlovian learning and performance with reciprocal associations. Psychological Review, 127, 829-852.), a model in which responses related to the nature of the unconditioned stimulus (e.g., goal-tracking) have a different origin than those related to the nature of the conditioned stimulus (e.g., sign-tracking)

HeiDI: A model for Pavlovian learning and performance with reciprocal associations

Associative treatments of how Pavlovian conditioning affects conditioned behavior are rudimentary: A simple ordinal mapping is held to exist between the strength of an association (V) between a conditioned stimulus (CS) and an unconditioned stimulus (US; i.e., VCS-US) and conditioned behavior in a given experimental preparation. The inadequacy of this simplification is highlighted by recent studies that have taken multiple measures of conditioned behavior: Different measures of conditioned behavior provide the basis for drawing opposite conclusions about VCS-US across individual animals. Here, we develop a simple model involving reciprocal associations between the CS and US (VCS-US and VUS-CS) that simulates these qualitative individual differences in conditioned behavior. The new model, HeiDI (How excitation and inhibition Determine Ideo-motion), enables a broad range of phenomena to be accommodated, which are either beyond the scope of extant models or require them to appeal to additional (learning) processes. It also provides an impetus for new lines of inquiry and generates novel predictions

Associative change in Pavlovian conditioning: a re-appraisal

Robert A. Rescorla changed how Pavlovian conditioning was studied and interpreted. His empirical contributions were fundamental and theoretically driven. One involved testing a central tenet of the model that he developed with Allan R. Wagner. The Rescorla-Wagner learning rule uses a pooled error term to determine changes in a directional association between the representations of the conditioned stimulus (CS) and unconditioned stimulus (US). This learning rule predicts that 2 equally salient CSs (A and B) will undergo equivalent associative change when they are conditioned in compound (i.e., AB→US). Rescorla’s results suggested that this was not the case (e.g., Rescorla, 2000). Here, we show that these results can be reconciled with a model that uses a learning rule with a pooled error term once that rule is applied equivalently to all of the stimuli presented on a given trial, and the resulting reciprocal associations (directly and indirectly) contribute to performance. This model, called HeiDI, integrates several features of Rescorla’s research and theorizing while addressing an issue that he recognized required further analysis: how learning is translated into performance

Prediction error in models of adaptive behavior

Pavlovian conditioning is evident in every species in which it has been assessed, and there is a consensus about its interpretation across behavioral,1,2 brain,3,4,5,6 and computational analyses7,8,9,10,11: conditioned behavior reflects the formation of a directional associative link from the memory of one stimulus (e.g., a visual stimulus) to another (e.g., food), with learning stopping when there is no error between the prediction generated by the visual stimulus and what happens next (e.g., food). This consensus fails to anticipate the results that we report here. In our experiments with rats, we find that arranging predictive (visual stimulus→food) and nonpredictive (food→visual stimulus) relationships produces marked and sustained changes in conditioned behaviors when the visual stimulus is presented alone. Moreover, the type of relationship affects (1) the distribution of conditioned behaviors related to the properties of both food (called goal-tracking) and the visual stimulus (called sign-tracking) and (2) when in the visual stimulus, these two behaviors are evident. These results represent an impetus for a fundamental shift in how Pavlovian conditioning is interpreted: animals learn about the relationship between two stimuli irrespective of the order in which they are presented, but they exhibit this knowledge in different ways. This interpretation and our new results are captured by a recent model of Pavlovian conditioning,12,13 HeiDI, and both are consistent with the need for animals to represent the fact that the impact of a cause (e.g., the ingestion of nutrients or the bite of a predator) can be felt before or after the cause has been perceived

Individual variation in the vigor and form of Pavlovian conditioned responses: analysis of a model system

Pavlovian conditioning results in individual variation in the vigor and form of acquired behaviors. Here, we describe a general-process model of associative learning (HeiDI; How excitation and inhibition determine ideo-motion) that provides an analysis for such variation together with a range of other important group-level phenomena. The model takes as its starting point the idea that pairings of a conditioned stimulus (CS) and an unconditioned stimulus (US) result in the formation of reciprocal associations between their central representations. The asymptotic values of these associations and the rate at which these are reached are held to be influenced by the perceived salience of the CS (αCS) and US (βUS). Importantly, whether this associative knowledge is exhibited in behavior that reflects the properties of the CS (e.g., sign-tracking) or US (e.g., goal-tracking) is also influenced by the relative values of αCS and βUS. In this way, HeiDI provides an analysis for both quantitative and qualitative individual differences generated by Pavlovian conditioning procedures