Motion planning and reactive control on learnt skill manifolds

We propose a novel framework for motion planning and control that is based on a manifold encoding of the desired solution set. We present an alternate, model-free, approach to path planning, replanning and control. Our approach is founded on the idea of encoding the set of possible trajectories as a skill manifold, which can be learnt from data such as from demonstration. We describe the manifold representation of skills, a technique for learning from data and a method for generating trajectories as geodesics on such manifolds. We extend the trajectory generation method to handle dynamic obstacles and constraints. We show how a state metric naturally arises from the manifold encoding and how this can be used for reactive control in an on-line manner. Our framework tightly integrates learning, planning and control in a computationally efficient representation, suitable for realistic humanoid robotic tasks that are defined by skill specifications involving high-dimensional nonlinear dynamics, kinodynamic constraints and non-trivial cost functions, in an optimal control setting. Although, in principle, such problems can be handled by well understood analytical methods, it is often difficult and expensive to formulate models that enable the analytical approach. We test our framework with various types of robotic systems – ranging from a 3-link arm to a small humanoid robot – and show that the manifold encoding gives significant improvements in performance without loss of accuracy. Furthermore, we evaluate the framework against a state-of-the-art imitation learning method. We show that our approach, by learning manifolds of robotic skills, allows for efficient planning and replanning in changing environments, and for robust and online reactive control

From visuomotor control to latent space planning for robot manipulation

Deep visuomotor control is emerging as an active research area for robot manipulation. Recent advances in learning sensory and motor systems in an end-to-end manner have achieved remarkable performance across a range of complex tasks. Nevertheless, a few limitations restrict visuomotor control from being more widely adopted as the de facto choice when facing a manipulation task on a real robotic platform. First, imitation learning-based visuomotor control approaches tend to suffer from the inability to recover from an out-of-distribution state caused by compounding errors. Second, the lack of versatility in task definition limits skill generalisability. Finally, the training data acquisition process and domain transfer are often impractical. In this thesis, individual solutions are proposed to address each of these issues. In the first part, we find policy uncertainty to be an effective indicator of potential failure cases, in which the robot is stuck in out-of-distribution states. On this basis, we introduce a novel uncertainty-based approach to detect potential failure cases and a recovery strategy based on action-conditioned uncertainty predictions. Then, we propose to employ visual dynamics approximation to our model architecture to capture the motion of the robot arm instead of the static scene background, making it possible to learn versatile skill primitives. In the second part, taking inspiration from the recent progress in latent space planning, we propose a gradient-based optimisation method operating within the latent space of a deep generative model for motion planning. Our approach bypasses the traditional computational challenges encountered by established planning algorithms, and has the capability to specify novel constraints easily and handle multiple constraints simultaneously. Moreover, the training data comes from simple random motor-babbling of kinematically feasible robot states. Our real-world experiments further illustrate that our latent space planning approach can handle both open and closed-loop planning in challenging environments such as heavily cluttered or dynamic scenes. This leads to the first, to our knowledge, closed-loop motion planning algorithm that can incorporate novel custom constraints, and lays the foundation for more complex manipulation tasks

Residual Learning from Demonstration: Adapting DMPs for Contact-rich Manipulation

Contacts and friction are inherent to nearly all robotic manipulation tasks. Through the motor skill of insertion, we study how robots can learn to cope when these attributes play a salient role. In this work we propose residual learning from demonstration (rLfD), a framework that combines dynamic movement primitives (DMP) that rely on behavioural cloning with a reinforcement learning (RL) based residual correction policy. The proposed solution is applied directly in task space and operates on the full pose of the robot. We show that rLfD outperforms alternatives and improves the generalisation abilities of DMPs. We evaluate this approach by training an agent to successfully perform both simulated and real world insertions of pegs, gears and plugs into respective sockets

A review on manipulation skill acquisition through teleoperation-based learning from demonstration

Manipulation skill learning and generalization have gained increasing attention due to the wide applications of robot manipulators and the spurt of robot learning techniques. Especially, the learning from demonstration method has been exploited widely and successfully in the robotic community, and it is regarded as a promising direction to realize the manipulation skill learning and generalization. In addition to the learning techniques, the immersive teleoperation enables the human to operate a remote robot with an intuitive interface and achieve the telepresence. Thus, it is a promising way to transfer manipulation skills from humans to robots by combining the learning methods and the teleoperation, and adapting the learned skills to different tasks in new situations. This review, therefore, aims to provide an overview of immersive teleoperation for skill learning and generalization to deal with complex manipulation tasks. To this end, the key technologies, e.g. manipulation skill learning, multimodal interfacing for teleoperation and telerobotic control, are introduced. Then, an overview is given in terms of the most important applications of immersive teleoperation platform for robot skill learning. Finally, this survey discusses the remaining open challenges and promising research topics

Acquisition and distribution of synergistic reactive control skills

Learning from demonstration is an afficient way to attain a new skill. In the context of autonomous robots, using a demonstration to teach a robot accelerates the robot learning process significantly. It helps to identify feasible solutions as starting points for future exploration or to avoid actions that lead to failure. But the acquisition of pertinent observationa is predicated on first segmenting the data into meaningful sequences. These segments form the basis for learning models capable of recognising future actions and reconstructing the motion to control a robot. Furthermore, learning algorithms for generative models are generally not tuned to produce stable trajectories and suffer from parameter redundancy for high degree of freedom robots This thesis addresses these issues by firstly investigating algorithms, based on dynamic programming and mixture models, for segmentation sensitivity and recognition accuracy on human motion capture data sets of repetitive and categorical motion classes. A stability analysis of the non-linear dynamical systems derived from the resultant mixture model representations aims to ensure that any trajectories converge to the intended target motion as observed in the demonstrations. Finally, these concepts are extended to humanoid robots by deploying a factor analyser for each mixture model component and coordinating the structure into a low dimensional representation of the demonstrated trajectories. This representation can be constructed as a correspondence map is learned between the demonstrator and robot for joint space actions. Applying these algorithms for demonstrating movement skills to robot is a further step towards autonomous incremental robot learning

Learning to represent surroundings, anticipate motion and take informed actions in unstructured environments

Contemporary robots have become exceptionally skilled at achieving specific tasks in structured environments. However, they often fail when faced with the limitless permutations of real-world unstructured environments. This motivates robotics methods which learn from experience, rather than follow a pre-defined set of rules. In this thesis, we present a range of learning-based methods aimed at enabling robots, operating in dynamic and unstructured environments, to better understand their surroundings, anticipate the actions of others, and take informed actions accordingly

Robot manipulator skill learning and generalising through teleoperation

Robot manipulators have been widely used for simple repetitive, and accurate tasks in industrial plants, such as pick and place, assembly and welding etc., but it is still hard to deploy in human-centred environments for dexterous manipulation tasks, such as medical examination and robot-assisted healthcare. These tasks are not only related to motion planning and control but also to the compliant interaction behaviour of robots, e.g. motion control, force regulation and impedance adaptation simultaneously under dynamic and unknown environments. Recently, with the development of collaborative robotics (cobots) and machine learning, robot skill learning and generalising have attained increasing attention from robotics, machine learning and neuroscience communities. Nevertheless, learning complex and compliant manipulation skills, such as manipulating deformable objects, scanning the human body and folding clothes, is still challenging for robots. On the other hand, teleoperation, also namely remote operation or telerobotics, has been an old research area since 1950, and there have been a number of applications such as space exploration, telemedicine, marine vehicles and emergency response etc. One of its advantages is to combine the precise control of robots with human intelligence to perform dexterous and safety-critical tasks from a distance. In addition, telepresence allows remote operators could feel the actual interaction between the robot and the environment, including the vision, sound and haptic feedback etc. Especially under the development of various augmented reality (AR), virtual reality (VR) and wearable devices, intuitive and immersive teleoperation have received increasing attention from robotics and computer science communities. Thus, various human-robot collaboration (HRC) interfaces based on the above technologies were developed to integrate robot control and telemanipulation by human operators for robot skills learning from human beings. In this context, robot skill learning could benefit teleoperation by automating repetitive and tedious tasks, and teleoperation demonstration and interaction by human teachers also allow the robot to learn progressively and interactively. Therefore, in this dissertation, we study human-robot skill transfer and generalising through intuitive teleoperation interfaces for contact-rich manipulation tasks, including medical examination, manipulating deformable objects, grasping soft objects and composite layup in manufacturing. The introduction, motivation and objectives of this thesis are introduced in Chapter 1. In Chapter 2, a literature review on manipulation skills acquisition through teleoperation is carried out, and the motivation and objectives of this thesis are discussed subsequently. Overall, the main contents of this thesis have three parts: Part 1 (Chapter 3) introduces the development and controller design of teleoperation systems with multimodal feedback, which is the foundation of this project for robot learning from human demonstration and interaction. In Part 2 (Chapters 4, 5, 6 and 7), we studied primitive skill library theory, behaviour tree-based modular method, and perception-enhanced method to improve the generalisation capability of learning from the human demonstrations. And several applications were employed to evaluate the effectiveness of these methods.In Part 3 (Chapter 8), we studied the deep multimodal neural networks to encode the manipulation skill, especially the multimodal perception information. This part conducted physical experiments on robot-assisted ultrasound scanning applications.Chapter 9 summarises the contributions and potential directions of this thesis. Keywords: Learning from demonstration; Teleoperation; Multimodal interface; Human-in-the-loop; Compliant control; Human-robot interaction; Robot-assisted sonography

구조로봇을 위한 강건한 계층적 동작 계획 및 제어

학위논문(박사) -- 서울대학교대학원 : 공과대학 기계항공공학부, 2021.8. 박종우.Over the last several years, robotics has experienced a striking development, and a new generation of robots has emerged that shows great promise in being able to accomplish complex tasks associated with human behavior. Nowadays the objectives of the robots are no longer restricted to the automaton in the industrial process but are changing into explorers for hazardous, harsh, uncooperative, and extreme environments. As these robots usually operate in dynamic and unstructured environments, they should be robust, adaptive, and reactive under various changing operation conditions. We propose online hierarchical optimization-based planning and control methodologies for a rescue robot to execute a given mission in such a highly unstructured environment. A large number of degrees of freedom is provided to robots in order to achieve diverse kinematic and dynamic tasks. However, accomplishing such multiple objectives renders on-line reactive motion planning and control problems more difficult to solve due to the incompatible tasks. To address this problem, we exploit a hierarchical structure to precisely resolve conflicts by creating a priority in which every task is achieved as much as possible according to the levels. In particular, we concentrate on the reasoning about the task regularization to ensure the convergence and robustness of a solution in the face of singularity. As robotic systems with real-time motion planners or controllers often execute unrehearsed missions, a desired task cannot always be driven to a singularity free configuration. We develop a generic solver for regularized hierarchical quadratic programming without resorting to any off-the-shelf QP solver to take advantage of the null-space projections for computational efficiency. Therefore, the underlying principles are thoroughly investigated. The robust optimal solution is obtained under both equality and inequality tasks or constraints while addressing all problems resulting from the regularization. Especially as a singular value decomposition centric approach is leveraged, all hierarchical solutions and Lagrange multipliers for properly handling the inequality constraints are analytically acquired in a recursive procedure. The proposed algorithm works fast enough to be used as a practical means of real-time control system, so that it can be used for online motion planning, motion control, and interaction force control in a single hierarchical optimization. Core system design concepts of the rescue robot are presented. The goals of the robot are to safely extract a patient and to dispose a dangerous object instead of humans. The upper body is designed humanoid in form with replaceable modularized dual arms. The lower body is featured with a hybrid tracked and legged mobile platform to simultaneously acquire versatile manipulability and all-terrain mobility. Thus, the robot can successfully execute a driving task, dangerous object manipulation, and casualty extraction missions by changing the pose and modularized equipments in an optimized manner. Throughout the dissertation, all proposed methods are validated through extensive numerical simulations and experimental tests. We highlight precisely how the rescue robot can execute a casualty extraction and a dangerous object disposal mission both in indoor and outdoor environments that none of the existing robots has performed.최근에 등장한 새로운 세대의 로봇은 기존에는 인간만이 할 수 있었던 복잡한 일을 로봇 또한 수행할 수 있음을 보여주었다. 특히 DARPA Robotics Challenge를 통해 이러한 사실을 잘 확인할 수 있으며, 이 로봇들은 공장과 같은 정형화된 환경에서 자동화된 일을 반복적으로 수행하던 임무에서 더 나아가 극한의 환경에서 인간을 대신하여 위험한 임무를 수행할 수 있는 방향으로 발전하고 있다. 그래서 사람들은 재난환경에서 안전하고 시의 적절하게 대응할 수 있는 여러 가지 대안 중에서 실현 가능성이 높은 대처 방안으로 로봇을 생각하게 되었다. 하지만 이러한 로봇은 동적으로 변화하는 비정형 환경에서 임무를 수행할 수 있어야 하기 때문에 불확실성에 대해 강건해야하고, 다양한 환경 조건에서 능동적으로 반응을 할 수 있어야 한다. 본 학위논문에서는 로봇이 비정형 환경에서 강건하면서도 적응적으로 동작할 수 있는 실시간 최적화 기반의 동작 계획 및 제어 방법과 구조 로봇의 설계 개념을 제안하고자 한다. 인간은 많은 자유도를 가지고 있으며, 하나의 전신 동작을 생성할 때 다양한 기구학 혹은 동역학적 특성을 가지는 세부 동작 혹은 작업을 정의하고, 이를 효과적으로 종합할 수 있다. 그리고 학습을 통해 각 동작 요소들을 최적화할 뿐만 아니라 상황 에 따라 각 동작 요소에 우선순위를 부여하여 이를 효과적으로 결합하거나 분리하여 실시간으로 최적의 동작을 생성하고 제어한다. 즉, 상황에 따라 중요한 동작요소를 우선적으로 수행하고 우선순위가 낮은 동작요소는 부분 혹은 전체적으로 포기하기도 하면서 매우 유연하게 전체 동작을 생성하고 최적화 한다. 인간과 같이 다자유도를 보유한 로봇 또한 기구학과 동역학적 특성을 가지는 다양한 세부 동작 혹은 작업을 작업공간(task space) 혹은 관절공간(configuration space)에서 정의할 수 있으며, 우선순위에 따라 이를 효과적으로 결합하여 전체 동작을 생 성하고 제어할 수 있다. 서로 양립하기 어려운 로봇의 동작 문제를 해결하기 위해 동작들 사이에 우선순위를 부여하여 계층을 생성하고, 이에 따라 로봇의 전신 동작을 구현하는 방법은 오랫동안 연구가 진행되어 왔다. 이러한 계층적 최적화를 이용하면 우선순위가 높은 동작부터 순차적으로 실행하지만, 우선순위가 낮은 동작요소들도 가능한 만족시키는 최적의 해를 찾을 수 있다. 하지만 관절의 구동 범위와 같은 부등식의 조건이 포함된 계층적 최적화 문제에서 특이점에 대한 강건성까지 확보할 수 있는 방법에 대해서는 아직까지 많은 부분이 밝 혀진 바가 없다. 따라서 본 학위논문에서는 등식과 부등식으로 표현되는 구속조건 혹은 동작요소를 계층적 최적화에 동시에 포함시키고, 특이점이 존재하더라도 강건성과 수렴성을 보장하는 관절공간에서의 최적해를 확보하는데 집중한다. 왜나하면 비정형 임무를 수행하는 로봇은 사전에 계획된 동작을 수행하는 것이 아닌 변화하는 환경조건에 따라 실시간으로 동작을 계획하고 제어해야 하기 때문에 특이점이 없는 자세로 로봇을 항상 제어하기가 어렵다. 그리고 이렇게 특이점을 회피하는 방향으로 로봇을 제어하는 것은 로봇의 운용성을 심각하게 저해시킬 수 있다. 특이점 근방에서의 해의 강건성이 보장되지 않으면 로봇 관절에 과도한 속도 혹은 토크가 발생하여 로봇의 임무 수행이 불가능하거나 환경과 로봇의 손상을 초래할 수 있으며, 나아가 로봇과 함께 임무를 수행하는 사람에게 상해를 가할 수도 있다. 특이점에 대한 강건성을 확보하기 위해 우선순위 기반의 계층적 최적화와 정규화 (regularization)를 통합하여 정규화된 계층적 최적화 (RHQP: Regularized Hierarchical Quadratic Program) 문제를 다룬다. 부등식이 포함된 계층적 최적화에 정규화를 동시에 고려함으로써 야기되는 많은 문제점들을 해결하고 해의 최적성과 강건성을 확보할 수 있는 방법을 제안한다. 특히 외부의 최적화 프로그램을 사용하지 않고 수치적 최적화 (numerical optimization) 이론과 우선순위에 기반을 두는 여유자유도 로봇의 해석 기법을 이용하여 계산의 효율성을 극대화할 수 있는 이차 프로그램(quadratic programming)을 제안한다. 또한 이와 동시에 정규화된 계층적 최적화 문제의 이론적 구조를 철저하게 분석한다. 특히 특이값 분해 (singular value decomposition)를 통해 최적해와 부등식 조건을 처리하는데 필요한 라그랑지 승수를 재귀적인 방법으로 해석적 형태로 구함으로써 계산의 효율성을 증대시키고 동시에 부등식의 조건을 오류 없이 정확하게 처리할 수 있도록 하였다. 그리고 정규화된 계층적 최적화를 힘제어까지 확장하여 환경과 로봇의 안전한 상호작용을 보장하여 로봇이 적절한 힘으로 환경과 접촉할 수 있도록 하였다. 불확실성이 존재하는 비정형 환경에서 비정형 임무를 수행할 수 있는 구조로봇의 핵심 설계 개념을 제시한다. 비정형 환경에서의 조작 성능과 이동 성능을 동시에 확보할 수 있는 형상으로 로봇을 설계하여 구조 로봇으로 하여금 최종 목적으로 설정된 인간을 대신하여 부상자를 구조하고 위험물을 처리하는 임무를 효과적으로 수행할 수 있도록 한다. 구조 로봇에 필요한 매니퓰레이터는 부상자 구조 임무와 위험물 처리 임무에 따라 교체 가능한 모듈형으로 설계하여 각각의 임무에 따라 최적화된 매니퓰 레이터를 장착하여 임무를 수행할 수 있다. 하체는 트랙과 관절이 결합된 하이브리드 형태를 취하고 있으며, 주행 임무와 조작임무에 따라 형상을 변경할 수 있다. 형상 변경과 모듈화된 매니퓰레이터를 통해서조작 성능과 험한 지형에서 이동할 수 있는 주행 성능을 동시에 확보하였다. 최종적으로 구조로봇의 설계와 실시간 계층적 제어를 이용하여 비정형 실내외 환경에서 구조로봇이 주행임무, 위험물 조작임무, 부상자 구조 임무를 성공적으로 수 행할 수 있음을 해석과 실험을 통하여 입증함으로써 본 학위논문에서 제안한 설계와 정규화된 계층적 최적화 기반의 제어 전략의 유용성을 검증하였다.1 Introduction 1 1.1 Motivations 1 1.2 Related Works and Research Problems for Hierarchical Control 3 1.2.1 Classical Approaches 3 1.2.2 State-of-the-Art Strategies 4 1.2.3 Research Problems 7 1.3 Robust Rescue Robots 9 1.4 Research Goals 12 1.5 Contributions of ThisThesis 13 1.5.1 Robust Hierarchical Task-Priority Control 13 1.5.2 Design Concepts of Robust Rescue Robot 16 1.5.3 Hierarchical Motion and ForceControl 17 1.6 Dissertation Preview 18 2 Preliminaries for Task-Priority Control Framework 21 2.1 Introduction 21 2.2 Task-Priority Inverse Kinematics 23 2.3 Recursive Formulation of Null Space Projector 28 2.4 Conclusion 31 3 Robust Hierarchical Task-Priority Control 33 3.1 Introduction 33 3.1.1 Motivations 35 3.1.2 Objectives 36 3.2 Task Function Approach 37 3.3 Regularized Hierarchical Optimization with Equality Tasks 41 3.3.1 Regularized Hierarchical Optimization 41 3.3.2 Optimal Solution 45 3.3.3 Task Error and Hierarchical Matrix Decomposition 49 3.3.4 Illustrative Examples for Regularized Hierarchical Optimization 56 3.4 Regularized Hierarchical Optimization with Inequality Constraints 60 3.4.1 Lagrange Multipliers 61 3.4.2 Modified Active Set Method 66 3.4.3 Illustrative Examples of Modified Active Set Method 70 3.4.4 Examples for Hierarchical Optimization with Inequality Constraint 72 3.5 DLS-HQP Algorithm 79 3.6 Concluding Remarks 80 4 Rescue Robot Design and Experimental Results 83 4.1 Introduction 83 4.2 Rescue Robot Design 85 4.2.1 System Design 86 4.2.2 Variable Configuration Mobile Platform 92 4.2.3 Dual Arm Manipulators 95 4.2.4 Software Architecture 97 4.3 Performance Verification for Hierarchical Motion Control 99 4.3.1 Real-Time Motion Generation 99 4.3.2 Task Specifications 103 4.3.3 Singularity Robust Task Priority 106 4.3.4 Inequality Constraint Handling and Computation Time 111 4.4 Singularity Robustness and Inequality Handling for Rescue Mission 117 4.5 Field Tests 122 4.6 Concluding Remarks 126 5 Hierarchical Motion and Force Control 129 5.1 Introduction 129 5.2 Operational Space Control 132 5.3 Acceleration-Based Hierarchical Motion Control 134 5.4 Force Control 137 5.4.1 Force Control with Inner Position Loop 141 5.4.2 Force Control with Inner Velocity Loop 144 5.5 Motion and Force Control 145 5.6 Numerical Results for Acceleration-Based Motion and Force Control 148 5.6.1 Task Specifications 150 5.6.2 Force Control Performance 151 5.6.3 Singularity Robustness and Inequality Constraint Handling 155 5.7 Velocity Resolved Motion and Force Control 160 5.7.1 Velocity-Based Motion and Force Control 161 5.7.2 Experimental Results 163 5.8 Concluding Remarks 167 6 Conclusion 169 6.1 Summary 169 6.2 Concluding Remarks 173 A Appendix 175 A.1 Introduction to PID Control 175 A.2 Inverse Optimal Control 176 A.3 Experimental Results and Conclusion 181 Bibliography 183 Abstract 207박

Working Inside the Black Box: Refinement of Pre-Existing Skills

This thesis aimed to address and inform the gap in current sport psychology/coaching research, knowledge and practice related to the implementation of technical refinement in already learnt, well-established and self-paced skills. This was achieved through a series of studies conducted within golf. Accordingly, Chapter 2 revealed technical refinement as neither systematic nor consistent within and between European Tour players and coaches and high-level amateurs. Building on this need, the systematic Five-A Model was derived from the literature (Chapter 3), targeting outcomes of permanency and pressure resistance. Following, motor control (Chapter 4) and kinematic (Chapter 5) measures, technological methods from which these data could be obtained (Chapter 6) and appropriate training environments and task characteristics (Chapter 7) were determined, aimed at enabling informative tracking of progress through the Five-A Model in applied golf coaching environments. Having developed these ranges of measures and methods, Chapter 8 presented three longitudinal case studies aimed at implementing and tracking progress through stages of the Five-A Model. Results revealed outcomes with different levels of success in facilitating technical refinement, based primarily on psycho-behavioural limitations that were also found in Chapter 2. Therefore, as a final check on measures proposed, Chapter 9 confirmed previous suggestions by tracking six performers making short-term technical refinements within a single training session. Finally, Chapter 10 summarised the findings and implications of this thesis. Particular emphasis was directed towards the impact of psycho-behavioural skills in determining the success when attempting refinements, the further development of informative measures to track progress and inform coaches decision making and the wider implications of this research within clinical and rehabilitation settings