Learning sequential motor tasks

Many real robot applications require the sequential use of multiple distinct motor primitives. This requirement implies the need to learn the individual primitives as well as a strategy to select the primitives sequentially. Such hierarchical learning problems are commonly either treated as one complex monolithic problem which is hard to learn, or as separate tasks learned in isolation. However, there exists a strong link between the robots strategy and its motor primitives. Consequently, a consistent framework is needed that can learn jointly on the level of the individual primitives and the robots strategy. We present a hierarchical learning method which improves individual motor primitives and, simultaneously, learns how to combine these motor primitives sequentially to solve complex motor tasks. We evaluate our method on the game of robot hockey, which is both difficult to learn in terms of the required motor primitives as well as its strategic elements

A Cognitive Model for Generalization during Sequential Learning

Traditional artificial neural network models of learning suffer from catastrophic interference. They are commonly trained to perform only one specific task, and, when trained on a new task, they forget the original task completely. It has been shown that the foundational neurocomputational principles embodied by the Leabra cognitive modeling framework, specifically fast lateral inhibition and a local synaptic plasticity model that incorporates both correlational and error-based components, are sufficient to largely overcome this limitation during the sequential learning of multiple motor skills. Evidence has also provided that Leabra is able to generalize the subsequences of motor skills, when doing so is appropriate. In this paper, we provide a detailed analysis of the extent of generalization possible with Leabra during sequential learning of multiple tasks. For comparison, we measure the generalization exhibited by the backpropagation of error learning algorithm. Furthermore, we demonstrate the applicability of sequential learning to a pair of movement tasks using a simulated robotic arm

Hybrid hierarchical learning for solving complex sequential tasks using the robotic manipulation network ROMAN

Solving long sequential tasks remains a non-trivial challenge in the field of embodied artificial intelligence. Enabling a robotic system to perform diverse sequential tasks with a broad range of manipulation skills is a notable open problem and continues to be an active area of research. In this work, we present a hybrid hierarchical learning framework, the robotic manipulation network ROMAN, to address the challenge of solving multiple complex tasks over long time horizons in robotic manipulation. By integrating behavioural cloning, imitation learning and reinforcement learning, ROMAN achieves task versatility and robust failure recovery. It consists of a central manipulation network that coordinates an ensemble of various neural networks, each specializing in different recombinable subtasks to generate their correct in-sequence actions, to solve complex long-horizon manipulation tasks. Our experiments show that, by orchestrating and activating these specialized manipulation experts, ROMAN generates correct sequential activations accomplishing long sequences of sophisticated manipulation tasks and achieving adaptive behaviours beyond demonstrations, while exhibiting robustness to various sensory noises. These results highlight the significance and versatility of ROMAN’s dynamic adaptability featuring autonomous failure recovery capabilities, and underline its potential for various autonomous manipulation tasks that require adaptive motor skills

RObotic MAnipulation Network (ROMAN) \unicode{x2013} Hybrid Hierarchical Learning for Solving Complex Sequential Tasks

Solving long sequential tasks poses a significant challenge in embodied artificial intelligence. Enabling a robotic system to perform diverse sequential tasks with a broad range of manipulation skills is an active area of research. In this work, we present a Hybrid Hierarchical Learning framework, the Robotic Manipulation Network (ROMAN), to address the challenge of solving multiple complex tasks over long time horizons in robotic manipulation. ROMAN achieves task versatility and robust failure recovery by integrating behavioural cloning, imitation learning, and reinforcement learning. It consists of a central manipulation network that coordinates an ensemble of various neural networks, each specialising in distinct re-combinable sub-tasks to generate their correct in-sequence actions for solving complex long-horizon manipulation tasks. Experimental results show that by orchestrating and activating these specialised manipulation experts, ROMAN generates correct sequential activations for accomplishing long sequences of sophisticated manipulation tasks and achieving adaptive behaviours beyond demonstrations, while exhibiting robustness to various sensory noises. These results demonstrate the significance and versatility of ROMAN's dynamic adaptability featuring autonomous failure recovery capabilities, and highlight its potential for various autonomous manipulation tasks that demand adaptive motor skills.Comment: To appear in Nature Machine Intelligence. Includes the main and supplementary manuscript. Total of 70 pages, with a total of 9 Figures and 17 Table

Impaired spatio-temporal predictive motor timing associated with spinocerebellar ataxia type 6

Many daily life activities demand precise integration of spatial and temporal information of sensory inputs followed by appropriate motor actions. This type of integration is carried out in part by the cerebellum, which has been postulated to play a central role in learning and timing of movements. Cerebellar damage due to atrophy or lesions may compromise forward- model processing, in which both spatial and temporal cues are used to achieve prediction for future motor states. In the present study we sought to further investigate the cerebellar contribution to predictive and reactive motor timing, as well as to learning of sequential order and temporal intervals in these tasks. We tested patients with spinocerebellar ataxia type 6 (SCA6) and healthy controls for two related motor tasks; one requiring spatio-temporal prediction of dynamic visual stimuli and another one requiring reactive timing only. We found that healthy controls established spatio-temporal prediction in their responses with high temporal precision, which was absent in the cerebellar patients. SCA6 patients showed lower predictive motor timing, coinciding with a reduced number of correct responses during the 'anticipatory' period on the task. Moreover, on the task utilizing reactive motor timing functions, control participants showed both sequence order and temporal interval learning, whereas patients only showed sequence order learning. These results suggest that SCA6 affects predictive motor timing and temporal interval learning. Our results support and highlight cerebellar contribution to timing and argue for cerebellar engagement during spatio-temporal prediction of upcoming events

Differential vulnerability of different forms of skill learning in Parkinson’s disease Different forms of skill learning in Parkinson’s disease

The striatal dopaminergic dysfunction in Parkinson’s disease (PD) has been associated with deficits in skill learning, but results are inconclusive so far. Motor sequence learning (especially sequence-specific learning) is found to be deficient in the majority of studies using the SRT task (Jackson, Jackson, Harrison, Henderson, & Kennard, 1995; Siegert, Taylor, Weatherall, & Abernethy, 2006). While problems with motor sequences seem to be prevalent, PD patients show intact performance on AGL tasks, suggesting that the sequencing problem may be response- or taskdependent (Reber & Squire, 1999). Acquisition of nonsequential probabilistic associations also seems to be vulnerable as evidenced by impaired probabilistic category learning performance in PD (Knowlton, Mangels, & Squire, 1996; Shohamy, Myers, Onlaor, & Gluck, 2004). Our aim was to explore the nature of the skill learning deficit by testing different types of skill learning (sequential versus nonsequential, motor versus verbal) in the same group of Parkinson’s patients. 34 patients with PD (mean age: 62.59.77 years, SD: 7.67) were compared to age-matched typical adults using 1) a Serial Reaction Time Task (SRT) testing the learning of motor sequences, 2) an Artificial Grammar Learning (AGL) task testing the extraction of regularities from auditory sequences and 3) a Weather prediction task (PCL-WP), testing probabilistic category learning in a non-sequential task. In motor sequence learning (SRT task), the two groups did not differ in accuracy; PD patients were generally slower, and analysis of z-transformed reaction times also revealed deficient motor sequence learning in PD compared to the control group. The PD group showed no evidence of sequence learning. The PD group showed the same amount of learning on the PCL task as controls, and we observed higher rates of learning on the AGL task in PD patients than in controls. These results support and also extend previous findings suggesting that motor skill learning is vulnerable in PD, while other forms of skill learning are less prone to impairment. Results are also in line with previous assumptions that mechanisms underlying artificial grammar learning and probabilistic categorization do not depend on the striatum (Reber & Squire, 1999)

Task integration in complex, bimanual sequence learning tasks

Sequence learning and multitasking studies have largely focused on simple motor skills, which cannot be directly transferred to the plethora of complex skills found outside of laboratory conditions. Established theories e.g. for bimanual tasks and task integration thus have to be reassessed in the context of complex motor skills. We hypothesize that under more complex conditions, task integration facilitates motor learning, impedes or suppresses effector-specific learning and can still be observed despite partial secondary task interference. We used the Ξ-apparatus to assess the learning success of six groups in a bimanual dual-task, in which we manipulated the degree of possible integration between the right-hand and the left-hand sequences. We could show that task integration positively influences the learning of these complex, bimanual skills. However, the integration impedes but not fully suppresses effector-specific learning, as we could measure reduced hand-specific learning. Task integration improves learning despite the disruptive effect of partial secondary task interference, but its mitigating effect is only effective to some extent. Overall, the results suggest that previous insights on sequential motor learning and task integration can largely also be applied to complex motor skills

Dual enhancement mechanisms for overnight motor memory consolidation

Our brains are constantly processing past events<sup>1</sup>. These offline processes consolidate memories, leading in the case of motor skill memories to an enhancement in performance between training sessions. A similar magnitude of enhancement develops over a night of sleep following an implicit task, in which a sequence of movements is acquired unintentionally, or following an explicit task, in which the same sequence is acquired intentionally<sup>2</sup>. What remains poorly understood, however, is whether these similar offline improvements are supported by similar circuits, or through distinct circuits. We set out to distinguish between these possibilities by applying transcranial magnetic stimulation over the primary motor cortex (M1) or the inferior parietal lobule (IPL) immediately after learning in either the explicit or implicit task. These brain areas have both been implicated in encoding aspects of a motor sequence and subsequently supporting offline improvements over sleep<sup>3,​4,​5</sup>. Here we show that offline improvements following the explicit task are dependent on a circuit that includes M1 but not IPL. In contrast, offline improvements following the implicit task are dependent on a circuit that includes IPL but not M1. Our work establishes the critical contribution made by M1 and IPL circuits to offline memory processing, and reveals that distinct circuits support similar offline improvements

Investigation of sequence processing: A cognitive and computational neuroscience perspective

Serial order processing or sequence processing underlies many human activities such as speech, language, skill learning, planning, problem-solving, etc. Investigating the neural bases of sequence processing enables us to understand serial order in cognition and also helps in building intelligent devices. In this article, we review various cognitive issues related to sequence processing with examples. Experimental results that give evidence for the involvement of various brain areas will be described. Finally, a theoretical approach based on statistical models and reinforcement learning paradigm is presented. These theoretical ideas are useful for studying sequence learning in a principled way. This article also suggests a two-way process diagram integrating experimentation (cognitive neuroscience) and theory/ computational modelling (computational neuroscience). This integrated framework is useful not only in the present study of serial order, but also for understanding many cognitive processes