Multi-modal Skill Memories for Online Learning of Interactive Robot Movement Generation

Queißer J. Multi-modal Skill Memories for Online Learning of Interactive Robot Movement Generation. Bielefeld: Universität Bielefeld; 2018.Modern robotic applications pose complex requirements with respect to the adaptation of actions regarding the variability in a given task. Reinforcement learning can optimize for changing conditions, but relearning from scratch is hardly feasible due to the high number of required rollouts. This work proposes a parameterized skill that generalizes to new actions for changing task parameters. The actions are encoded by a meta-learner that provides parameters for task-specific dynamic motion primitives. Experimental evaluation shows that the utilization of parameterized skills for initialization of the optimization process leads to a more effective incremental task learning. A proposed hybrid optimization method combines a fast coarse optimization on a manifold of policy parameters with a fine-grained parameter search in the unrestricted space of actions. It is shown that the developed algorithm reduces the number of required rollouts for adaptation to new task conditions. Further, this work presents a transfer learning approach for adaptation of learned skills to new situations. Application in illustrative toy scenarios, for a 10-DOF planar arm, a humanoid robot point reaching task and parameterized drumming on a pneumatic robot validate the approach. But parameterized skills that are applied on complex robotic systems pose further challenges: the dynamics of the robot and the interaction with the environment introduce model inaccuracies. In particular, high-level skill acquisition on highly compliant robotic systems such as pneumatically driven or soft actuators is hardly feasible. Since learning of the complete dynamics model is not feasible due to the high complexity, this thesis examines two alternative approaches: First, an improvement of the low-level control based on an equilibrium model of the robot. Utilization of an equilibrium model reduces the learning complexity and this thesis evaluates its applicability for control of pneumatic and industrial light-weight robots. Second, an extension of parameterized skills to generalize for forward signals of action primitives that result in an enhanced control quality of complex robotic systems. This thesis argues for a shift in the complexity of learning the full dynamics of the robot to a lower dimensional task-related learning problem. Due to the generalization in relation to the task variability, online learning for complex robots as well as complex scenarios becomes feasible. An experimental evaluation investigates the generalization capabilities of the proposed online learning system for robot motion generation. Evaluation is performed through simulation of a compliant 2-DOF arm and scalability to a complex robotic system is demonstrated for a pneumatically driven humanoid robot with 8-DOF

Bootstrapping of parameterized skills through hybrid optimization in task and policy spaces

Queißer J, Steil JJ. Bootstrapping of parameterized skills through hybrid optimization in task and policy spaces. Frontiers in Robotics and AI. 2018;5:49.Modern robotic applications create high demands on adaptation of actions with respect to variance in a given task. Reinforcement learning is able to optimize for these changing conditions, but relearning from scratch is hardly feasible due to the high number of required rollouts. We propose a parameterized skill that generalizes to new actions for changing task parameters, which is encoded as a meta-learner that provides parameters for task-specific dynamic motion primitives. Our work shows that utilizing parameterized skills for initialization of the optimization process leads to a more effective incremental task learning. In addition, we introduce a hybrid optimization method that combines a fast coarse optimization on a manifold of policy parameters with a fine grained parameter search in the unrestricted space of actions. The proposed algorithm reduces the number of required rollouts for adaptation to new task conditions. Application in illustrative toy scenarios, for a 10-DOF planar arm, and a humanoid robot point reaching task validate the approach

Learning the end-effector pose from demonstration for the Bionic Handling Assistant robot

Malekzadeh M, Queißer J, Steil JJ. Learning the end-effector pose from demonstration for the Bionic Handling Assistant robot. Presented at the 9th Int. Workshop on Human-Friendly Robotics, Genoa

Incremental Bootstrapping of Parameterized Motor Skills

Queißer J, Reinhart F, Steil JJ. Incremental Bootstrapping of Parameterized Motor Skills. In: Proc. IEEE Humanoids. IEEE; 2016.Many motor skills have an intrinsic, low-dimensional parameterization, e.g. reaching through a grid to different targets. Repeated policy search for new parameterizations of such a skill is inefficient, because the structure of the skill variability is not exploited. This issue has been previously addressed by learning mappings from task parameters to policy parameters. In this work, we introduce a bootstrapping technique that establishes such parameterized skills incrementally. The approach combines iterative learning with state-of-the-art black-box policy optimization. We investigate the benefits of incrementally learning parameterized skills for efficient policy retrieval and show that the number of required rollouts can be significantly reduced when optimizing policies for novel tasks. The approach is demonstrated for several parameterized motor tasks including upper-body reaching motion generation for the humanoid robot COMAN

Adaptive Handling Assistance for Industrial Lightweight Robots in Simulation

Balayn A, Queißer J, Wojtynek M, Wrede S. Adaptive Handling Assistance for Industrial Lightweight Robots in Simulation. In: 2016 IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR). 2016.With the growth of lightweight robots in industry handling assistance becomes more and more important for solving industrial tasks. We present an adaptive compliance control mode for industrial robots based on a learned equilibrium model. This enables us to cope with changing manufacturing environments, attached grippers as well as devices with inaccurate dynamic models, e.g. stiff tubes, wires, protection shields or hose packages. A further feature of the proposed method is the expandability by an additional parameterization that allows to deal with task variability. In this work we evaluate our approach using the example of a task that incorporates variable payloads. All experiments are conducted in a simulation framework to evaluate the feasibility of the proposed approach for industrial robot scenarios

Transfer Learning of Complex Motor Skills on the Humanoid Robot Affetto

Schulz A, Queißer J, Ishihara H, Asada M. Transfer Learning of Complex Motor Skills on the Humanoid Robot Affetto. Presented at the International Conference on Development and Learning and on Epigenetic Robotics 2018 (ICDL-EPIROB2018), Tokyo (In Press)

Imitation learning for a continuum trunk robot

Malekzadeh M, Queißer J, Steil JJ. Imitation learning for a continuum trunk robot. In: Verleysen M, ed. Proceedings of the 25. European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning. ESANN 2017. Louvain-la-Neuve: Ciaco; 2017

Emergence of Content-Agnostic Information Processing by a Robot Using Active Inference, Visual Attention, Working Memory, and Planning

Generalization by learning is an essential cognitive competency for humans. For example, we can manipulate even unfamiliar objects and can generate mental images before enacting a preplan. How is this possible? Our study investigated this problem by revisiting our previous study (Jung, Matsumoto, & Tani, 2019), which examined the problem of vision-based, goal-directed planning by robots performing a task of block stacking. By extending the previous study, our work introduces a large network comprising dynamically interacting submodules, including visual working memory (VWMs), a visual attention module, and an executive network. The executive network predicts motor signals, visual images, and various controls for attention, as well as masking of visual information. The most significant difference from the previous study is that our current model contains an additional VWM. The entire network is trained by using predictive coding and an optimal visuomotor plan to achieve a given goal state is inferred using active inference. Results indicate that our current model performs significantly better than that used in Jung et al. (2019), especially when manipulating blocks with unlearned colors and textures. Simulation results revealed that the observed generalization was achieved because content-agnostic information processing developed through synergistic interaction between the second VWM and other modules during the course of learning, in which memorizing image contents and transforming them are dissociated. This letter verifies this claim by conducting both qualitative and quantitative analysis of simulation results

Skill Memories for Parameterized Dynamic Action Primitives on the Pneumatically Driven Humanoid Robot Child Affetto

Queißer J, Ishihara H, Hammer B, Steil JJ, Asada M. Skill Memories for Parameterized Dynamic Action Primitives on the Pneumatically Driven Humanoid Robot Child Affetto. Presented at the International Conference on Development and Learning and on Epigenetic Robotics 2018 (ICDL-EPIROB2018), Tokyo

An Active Compliant Control Mode for Interaction with a Pneumatic Soft Robot

Queißer J, Neumann K, Rolf M, Reinhart F, Steil JJ. An Active Compliant Control Mode for Interaction with a Pneumatic Soft Robot. In: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2014). IEEE; 2014: 573-579.Bionic soft robots offer exciting perspectives for more flexible and safe physical interaction with the world and humans. Unfortunately, their hardware design often prevents analytical modeling, which in turn is a prerequisite to apply classical automatic control approaches. On the other hand, also modeling by means of learning is hardly feasible due to many degrees of freedom, high-dimensional state spaces and the softness properties like e.g. mechanical elasticity, which cause limited repeatability and complex dynamics. Nevertheless, the realization of basic control modes is important to leverage the potential of soft robots for applications. We therefore propose a hybrid approach combining classical and learning elements for the realization of an interactive control mode for an elastic bionic robot. It superimposes a low-gain feedback control with a feed-forward control based on a learned simplified model of the inverse dynamics which considers only equilibria of the robot’s dynamics. We demonstrate on the Bionic Handling Assistant how a respective inverse equilibrium model can be learned and effectively exploited for quick and agile control. In a second step, the control scheme is extended to an active compliant control mode. It implements a kind of gravitation compensation to allow for kinesthetic teaching of the robot based on the implicit knowledge of gravitational and mechanical forces that are encoded in the learned equilibrium model.We finally discuss that this control scheme may be implemented also on other soft robots to provide the avenue towards their applications in general manipulation tasks