Short and long term plasticity as cause-effect hypothesis testing in robotic ambiguous scenario

Andrea Soltoggio (1248822)

English

Loughborough University Institutional Repository

Short and long term plasticity as cause-effect hypothesis testing in robotic ambiguous scenariosAndrea SoltoggioNeurorobotic systems learn with difficulties when a continuous flow of information and delays make the cause-effect relationships ambiguous. The Hypothesis Testing (HT) plasticity proposed in this study models learning dynamics that account for ambiguity in the sensory-motor information flow improving drastically discrimination capabilities and memory capacity with respect to previous models. The new rule models consolidation to long-term memory and helps solve the plasticity-stability dilemma. References. - Soltoggio, Reinhart, Lemme, Steil. Learning the rules of a game: neural conditioning in human-robot interaction with delayed rewards. ICDL 2013- Soltoggio, Lemme, Reinhart, Steil. Rare neural correlations implement robotic conditioning with delayed rewards and disturbances. Frontiers in Neurorobotics (2013)Conclusion. The HT-plasticity improves both exploration and exploitation capabilities of the network. It increases memory capabilities by preserving established learned relationships, it detects coincidental facts as irrelevant and allows for the combination of more learning scenarios and skills.A BC Dhappy sadrock scissorsscissorspapertouch to right armactivetouchsensorsactivetouchsensorstouch to left armiCub plays Rock-Paper-ScissorsHT-plasticity in a network modelSIMULATIONSAmbiguous cause-effect relationshipswith delayed rewardsWhat sequence of stimuliand actions effectivelycauses a reward? One singleinstance cannot disambiguatethe relationship if delays and asynchronous informationflow are present.timestimuli actionss273s133s98s22s211s150s59s240a25a10a27a3a5a14a16a9agentdelayedreward1 sRarely Correlating Hebbian Plasticity (RCHP)(equivalent to R-STDP)eligibility tracesweight update  wji(t) = cji(t) · d(t)RCHPji(t) =8<: +1 if vj(t  tpt) · vi(t) > ✓hi 1 if vj(t  tpt) · vi(t) < ✓lo0 otherwisecji(t+ t) = cji(t) · e  t⌧c +RCHPji(t) .Hypothesis Testing Plasticity. Addition of 1) short-term and long-term components2) hypothesis testing potentiation and depression with negative baseline modulation (term b)eligibility tracesshort-term weight dynamicsRCHP+ji(t) =⇢+1 if vj(t  tpt) · vi(t) > ✓hi0 otherwisecji(t+ t) = cji(t) · e  t⌧c +RCHP+ji(t) .w˙stji(t) =  wstji/⌧st +m(t) · cji(t)Wji(t) = wstji(t) + wltji(t)w˙ltji(t) = ⇢ ·H(wstji(t)  )Consolidation ofhypotheses (weights)S2S3S4S5S300S1300 inputs 30 outputs. . .. . .Action 1Action 2Action 3Action 30random input flowdelayed reward / modulationeligibilitytracescompeting actionsfeedback to networkplastic weightsshort-term + long-termcomponentsRCHP ruleHT rulea b12 24 36 48050100time (h)reward / hwith RCHP rulepreviouslylearnedscenario 1scenario 3RCHP rule12 24 36 48050100time (h)reward / h    with HT rulepreviouslylearnedscenario 1scenario 3 HT ruleNEURO-ROBOTICS APPLICATIONS : combination of skills48 72 960246810time (h)cumulative short−term weightHT−Sc3RCHP−Sc1HT−Sc1RCHP−Sc30 0.5 101234synapses count (log)weight strengthreward−inducingsynapsesRCHP − end of scenario 1(0−24h)0 0.5 101234weight strengthsynapses count (log)reward−inducingsynapsesHT − end of scenario 1(0−24h)0 0.5 101234weight strengthRCHP − end of scenario 1revisiting (72−96h)reward−inducingsynapsesa0 0.5 101234weight strengthHT − end of scenario 1 revisiting (72−96h)reward−inducingsynapsesb(a) Memory capacity, preservation of information over different learning scenarios (b) Utility of memory when revisiting a previously learned scenarioiCub learns the colorsUncertain feedback signals in human-robot interactionOperant conditioning  with reversal learning. Learning from trial and error in human-robot interactionsClassical conditioning: iCub makes new friends(a) Improved exploration speed  (b) Discrimination capability and robustness of the HT rule with respect to the previous approachesProblem- large input-output flow- delayed rewards- ambiguous cause-effect relationships- no external reward-predictors (i.e. expected average rewards)Network- 300 inputs, 30 outputs- 9000 stimulus-action pairsAdvantages- short-term components represent hypotheses- long-term components represent established facts- HT allows for learning over multiple scenarios- HT plasticity allows for higher discrimination of true cause-effect relationshipsSuitable for- using memory capacity of network- modeling knowledge-discovering in ambiguous environments- studying classical and operant conditioning over long time scale and multiple associationsd(t+ t) = d(t) · e  t⌧m +   r(t) + b .modulationPrevious experiments with RCHP can now be all combined and expanded thanks to the increased memory capabilities and different learning scenarios

Short and long term plasticity as cause-effect hypothesis testing in robotic ambiguous scenarios

https://repository.lboro.ac.uk/articles/Short_and_long_term_plasticity_as_cause-effect_hypothesis_testing_in_robotic_ambiguous_scenarios/9404855/files/17021564.pdf

Short and long term plasticity as cause-effect hypothesis testing in robotic ambiguous scenarios

Abstract

Similar works

Full text

Available Versions

Loughborough University Institutional Repository