Accounting for negative automaintenance in pigeons: a dual learning systems approach and factored representations.

Animals, including Humans, are prone to develop persistent maladaptive and suboptimal behaviours. Some of these behaviours have been suggested to arise from interactions between brain systems of Pavlovian conditioning, the acquisition of responses to initially neutral stimuli previously paired with rewards, and instrumental conditioning, the acquisition of active behaviours leading to rewards. However the mechanics of these systems and their interactions are still unclear. While extensively studied independently, few models have been developed to account for these interactions. On some experiment, pigeons have been observed to display a maladaptive behaviour that some suggest to involve conflicts between Pavlovian and instrumental conditioning. In a procedure referred as negative automaintenance, a key light is paired with the subsequent delivery of food, however any peck towards the key light results in the omission of the reward. Studies showed that in such procedure some pigeons persisted in pecking to a substantial level despite its negative consequence, while others learned to refrain from pecking and maximized their cumulative rewards. Furthermore, the pigeons that were unable to refrain from pecking could nevertheless shift their pecks towards a harmless alternative key light. We confronted a computational model that combines dual-learning systems and factored representations, recently developed to account for sign-tracking and goal-tracking behaviours in rats, to these negative automaintenance experimental data. We show that it can explain the variability of the observed behaviours and the capacity of alternative key lights to distract pigeons from their detrimental behaviours. These results confirm the proposed model as an interesting tool to reproduce experiments that could involve interactions between Pavlovian and instrumental conditioning. The model allows us to draw predictions that may be experimentally verified, which could help further investigate the neural mechanisms underlying theses interactions

Prediction of the model about expected patterns of dopaminergic activity in negative automaintenance.

<p>Data are expressed as mean ± SEM. Average RPE computed by the FMF system at CS appearance (red) and removal of the CS after engagement with the negative key light (no US; gray) and withholding (US; black) for each session of conditioning in the whole population of pigeons (STs and GTs).</p

Computational representation of the negative automaintenance procedure.

<p>MDP accounting for Experiment 1 in Williams and Williams <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0111050#pone.0111050-Williams1" target="_blank">[8]</a> and for the Brief PA protocol of Sanabria et al. <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0111050#pone.0111050-Sanabria1" target="_blank">[19]</a>. States are described by a set of variables: <i>K</i>/<i>F</i> - negative Key light/Food is available (Magazine is always available, hence it is not shown), <i>cM</i>/<i>cK</i> - close to the Magazine/negative Key light, <i>Ka</i> - Key light appearance. The initial state is double circled, the dashed state is terminal and terminates the current episode. Actions are engage (eng) or refrain from engaging (<i>ngo</i>) with the proximal stimuli, explore (exp), or <i>go</i> to the <i>M</i>agazine/<i>K</i>ey light and <i>eat</i>. Only the <i>eat</i> action is rewarded (R), such that in this experiment, pigeons that engage with the key light receive nothing during the trial. For each action, the feature being focused on is displayed within brackets.</p

scattered

scattered p pplWe crept along slowly for several days, picking up a scattered seal here and there, but not many.PRINTED ITEM DNE-citW. KIRWIN, 2/75 JH 2/75Used I and SupUsed I and Sup2Used Iscatter, scattering, numerou

MDP for simulation of Experiment 4 of Williams and Williams.

<p>Legend is as in <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0111050#pone-0111050-g003" target="_blank">Figure 3</a>. A new continuous irrelevant key light (purple), the associated paths and actions are added to MDP of <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0111050#pone-0111050-g003" target="_blank">Figure 3</a> (Block A). Note that while not shown, as for the Magazine, the Continuous key light is present in all states. Paths are activated/deactivated depending on the current phase of the current protocol (<a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0111050#pone-0111050-t001" target="_blank">Table 1</a>).</p

Simulation of Experiment 1 of Williams and Williams [8] and Brief PA protocol of Sanabria et al. [19].

<p>(<b>A</b>) Cumulative pecks towards negative key light made by 8 simulated GT pigeons (blue curve) and 8 simulated ST pigeons (red curve). The dotted grey curve simulated the worse case scenario (if pigeons would have pecked at every trials). Data are expressed as mean ± SEM. (<b>B</b>) Zoom of (A) for a better reading of the blue curve (GTs). (<b>C</b>) Cumulative pecks for one ST pigeon by blocks of 50 trials. To be paralleled with <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0111050#pone-0111050-g001" target="_blank">Figure 1</a> of <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0111050#pone.0111050-Williams1" target="_blank">[8]</a>. (<b>D</b>) Cumulative pecks for one GT pigeon by blocks of 50 trials.</p

Experimental setups for Experiment 4.

<p>Experimental setups for Experiment 4.</p

Modelling individual differences in the form of Pavlovian conditioned approach responses: a dual learning systems approach with factored representations.

Reinforcement Learning has greatly influenced models of conditioning, providing powerful explanations of acquired behaviour and underlying physiological observations. However, in recent autoshaping experiments in rats, variation in the form of Pavlovian conditioned responses (CRs) and associated dopamine activity, have questioned the classical hypothesis that phasic dopamine activity corresponds to a reward prediction error-like signal arising from a classical Model-Free system, necessary for Pavlovian conditioning. Over the course of Pavlovian conditioning using food as the unconditioned stimulus (US), some rats (sign-trackers) come to approach and engage the conditioned stimulus (CS) itself - a lever - more and more avidly, whereas other rats (goal-trackers) learn to approach the location of food delivery upon CS presentation. Importantly, although both sign-trackers and goal-trackers learn the CS-US association equally well, only in sign-trackers does phasic dopamine activity show classical reward prediction error-like bursts. Furthermore, neither the acquisition nor the expression of a goal-tracking CR is dopamine-dependent. Here we present a computational model that can account for such individual variations. We show that a combination of a Model-Based system and a revised Model-Free system can account for the development of distinct CRs in rats. Moreover, we show that revising a classical Model-Free system to individually process stimuli by using factored representations can explain why classical dopaminergic patterns may be observed for some rats and not for others depending on the CR they develop. In addition, the model can account for other behavioural and pharmacological results obtained using the same, or similar, autoshaping procedures. Finally, the model makes it possible to draw a set of experimental predictions that may be verified in a modified experimental protocol. We suggest that further investigation of factored representations in computational neuroscience studies may be useful

The eSpiro Ventilator: An Open-Source Response to a Worldwide Pandemic

International audienceObjective: To address the issue of ventilator shortages, our group (eSpiro Network) developed a freely replicable, open-source hardware ventilator. Design: We performed a bench study. Setting: Dedicated research room as part of an ICU affiliated to a university hospital. Subjects: We set the lung model with three conditions of resistance and linear compliance for mimicking different respiratory mechanics of representative intensive care unit (ICU) patients. Interventions: The performance of the device was tested using the ASL5000 lung model. Measurements and Main Results: Twenty-seven conditions were tested. All the measurements fell within the ±10% limits for the tidal volume (VT). The volume error was influenced by the mechanical condition (p = 5.9 × 10−15) and the PEEP level (P = 1.1 × 10−12) but the clinical significance of this finding is likely meaningless (maximum −34 mL in the error). The PEEP error was not influenced by the mechanical condition (p = 0.25). Our experimental results demonstrate that the eSpiro ventilator is reliable to deliver VT and PEEP accurately in various respiratory mechanics conditions. Conclusions: We report a low-cost, easy-to-build ventilator, which is reliable to deliver VT and PEEP in passive invasive mechanical ventilation

Systems combined in the model and the variants.

<p>Variants of the model rely on the same architecture (described in <a href="http://www.ploscompbiol.org/article/info:doi/10.1371/journal.pcbi.1003466#pcbi-1003466-g002" target="_blank">Figure 2</a>) and only differ in the combined systems. Colours are shared for similar systems. (<b>A</b>) The model combines a Model-Based system (MB, in blue) and a Feature-Model-Free (FMF, in red) system. (<b>B</b>) Variant 1 combines a Model-Free system (MF, in green) and a Feature-Model-Free system. (<b>C</b>) Variant 2 combines a Model-Free system and a Bias system (BS, in grey), that relies on values from the Model-Free system. (<b>D</b>) Variant 3 combines a Model-Free system and two Bias systems, that rely on values from the Model-Free system. Variant 4 is not included as it failed to even reproduce the autoshaping behavioural results.</p