Robot Introspection with Bayesian Nonparametric Vector Autoregressive Hidden Markov Models

Robot introspection, as opposed to anomaly detection typical in process monitoring, helps a robot understand what it is doing at all times. A robot should be able to identify its actions not only when failure or novelty occurs, but also as it executes any number of sub-tasks. As robots continue their quest of functioning in unstructured environments, it is imperative they understand what is it that they are actually doing to render them more robust. This work investigates the modeling ability of Bayesian nonparametric techniques on Markov Switching Process to learn complex dynamics typical in robot contact tasks. We study whether the Markov switching process, together with Bayesian priors can outperform the modeling ability of its counterparts: an HMM with Bayesian priors and without. The work was tested in a snap assembly task characterized by high elastic forces. The task consists of an insertion subtask with very complex dynamics. Our approach showed a stronger ability to generalize and was able to better model the subtask with complex dynamics in a computationally efficient way. The modeling technique is also used to learn a growing library of robot skills, one that when integrated with low-level control allows for robot online decision making.Comment: final version submitted to humanoids 201

Understanding of Object Manipulation Actions Using Human Multi-Modal Sensory Data

Object manipulation actions represent an important share of the Activities of Daily Living (ADLs). In this work, we study how to enable service robots to use human multi-modal data to understand object manipulation actions, and how they can recognize such actions when humans perform them during human-robot collaboration tasks. The multi-modal data in this study consists of videos, hand motion data, applied forces as represented by the pressure patterns on the hand, and measurements of the bending of the fingers, collected as human subjects performed manipulation actions. We investigate two different approaches. In the first one, we show that multi-modal signal (motion, finger bending and hand pressure) generated by the action can be decomposed into a set of primitives that can be seen as its building blocks. These primitives are used to define 24 multi-modal primitive features. The primitive features can in turn be used as an abstract representation of the multi-modal signal and employed for action recognition. In the latter approach, the visual features are extracted from the data using a pre-trained image classification deep convolutional neural network. The visual features are subsequently used to train the classifier. We also investigate whether adding data from other modalities produces a statistically significant improvement in the classifier performance. We show that both approaches produce a comparable performance. This implies that image-based methods can successfully recognize human actions during human-robot collaboration. On the other hand, in order to provide training data for the robot so it can learn how to perform object manipulation actions, multi-modal data provides a better alternative

Human skill capture: A hidden Markov model of force and torque data in peg-in-a-hole assembly process

A new model has been constructed to generalise the force and torque information during a manual peg-in-a-hole (PiH) assembly process. The paper uses Hidden Markov Model analysis to interpret the state topology (transition probability) and observations (force/torque signal) in the manipulation task. The task can be recognised as several discrete states that reflect the intrinsic nature of the process. Since the whole manipulation process happens so fast, even the operator themselves cannot articulate the exact states. Those are tacit skills which are difficult to extract using human factors methodologies. In order to programme a robot to complete tasks at skill level, numerical representation of the sub-goals are necessary. Therefore, those recognised ‘hidden’ states become valuable when a detail explanation of the task is needed and when a robot controller needs to change its behaviour in different states. The Gaussian Mixture model (GMM) is used as the initial guess of observations distribution. Then a Hidden Markov Model is used to encode the state (sub-goal) topology and observation density associated with those sub-goals. The Viterbi algorithm is then applied for the model-based analysis of the force and torque signal and the classification into sub-goals. The Baum-Welch algorithm is used for training and to estimate the most likely model parameters. In addition to generic states recognition, the proposed method also enhances our understanding of the skill based performances in manual tasks

불확실성을 포함하는 조립작업을 위한 컴플라이언스 기반 펙인홀 전략

학위논문 (박사) -- 서울대학교 대학원 : 융합과학기술대학원 융합과학부(지능형융합시스템전공), 2020. 8. 박재흥.The peg-in-hole assembly is a representative robotic task that involves physical contact with the external environment. The strategies generally involve performing the assembly task by estimating the contact state between the peg and the hole. The contact forces and moments, measured using force sensors, are primarily used to estimate the contact state. In this paper, in contrast to past research in the area, which has involved the utilization of such expensive devices as force/torque sensors or remote compliance mechanisms, an inexpensive method is proposed for peg-in-hole assembly without force feedback or passive compliance mechanisms. The method consists of an analysis of the state of contact between the peg and the hole as well as a strategy to overcome the inevitable positional uncertainty of the hole incurred in the recognition process. A control scheme was developed to yield compliant behavior from the robot with physical contact under the condition of hybrid position/force control. Proposed peg-in-hole strategy is based on compliance characteristics and generating the force and moment. The peg is inserted into the hole as it adapts to the external environment. The effectiveness of the proposed method was experimentally verified using a humanoid upper body robot with fifty degrees of freedom and a peg-in-hole apparatus with a small clearance (0.1 mm). Three cases of experiments were conducted; Assembling the peg attached to the arm in the hole fixed in the external environment, grasping a peg with an anthropomorphic hand and assembling it into a fixed hole, and grasping both peg and hole with both hands and assembling each other. In order to assemble the peg-in-hole through the proposed strategy by the humanoid upper body robot, I present a method of gripping an object, estimating the kinematics of the gripped object, and manipulating the gripped object. In addition to the cost aspect, which is the fundamental motivation for the proposed strategy, the experimental results show that the proposed strategy has advantages such as fast assembly time and high success rate, but has the disadvantage of unpredictable elapsed time. The reason for having a high variance value for the success time is that the spiral trajectory, which is most commonly used, is used. In this study, I analyze the efficiency of spiral force trajectory and propose an improved force trajectory. The proposed force trajectory reduces the distribution of elapsed time by eliminating the uncertainty in the time required to find a hole. The efficiency of the force trajectory is analyzed numerically, verified through repeated simulations, and verified by the actual experiment with humanoid upper body robot developed by Korea institute of industrial technology.펙인홀 조립은 로봇의 접촉 작업을 대표하는 작업으로, 펙인홀 조립 전략을 연구함으로써 산업 생산 분야의 조립작업에 적용할 수 있다. 펙인홀 조립작업은 일반적으로 펙과 홀 간의 접촉상태를 추정함으로써 이루어진다. 접촉상태를 추정하기 위해 가장 널리 쓰이는 방법은 힘 센서를 사용하는 것인데, 접촉 힘과 모멘트를 측정하여 접촉상태를 추정하는 방식이다. 만약 이러한 센서를 사용하지 않을 수 있다면, 하드웨어 비용과 소프트웨어 연산량 감소 등의 장점이 있음은 자명하다. 본 논문에서는 힘 센서 혹은 수동 컴플라이언스 장치를 사용하지 않는 펙인홀 전략을 제안한다. 홀에 대한 인식 오차 혹은 로봇의 제어 오차를 극복하기 위하여 먼저 펙과 홀의 접촉 가능 상태를 분석하고 로봇의 컴플라이언스 모션을 위한 제어 프레임워크를 디자인한다. 전략은 컴플라이언스 특징에 기반하며 펙에 힘과 모멘트를 생성시킴으로써 조립작업을 수행한다. 펙은 외부환경에 순응함으로써 홀에 삽입된다. 제안한 전략은 낮은 공차를 갖는 펙인홀 실험을 통해서 그 유효성이 검증된다. 펙과 홀을 로봇팔과 외부환경에 각각 고정된 환경에서의 실험, 인간형 로봇핸드를 이용하여 펙을 잡아서 고정된 홀에 삽입하는 실험, 그리고 테이블에 놓인 펙과 홀을 각각 로봇핸드로 파지하여 조립하는 총 세 가지의 실험을 수행하였다. 핸드로 펙을 파지하고 조작하기 위하여, 파지 방법과 핸드를 이용한 물체 조작 알고리즘을 간략히 소개하였다. 제안한 전략의 성능을 실험적으로 분석한 결과, 높은 조립 성공률을 갖는 대신 조립시간이 예측할 수 없는 단점이 나타나 이를 보완하기 위해서 렌치 궤적 또한 제안하였다. 먼저 가장 일반적으로 사용되는 나선 힘 궤적을 이용했을 때 조립 성공시간의 분산이 큰 이유를 확률개념을 이용해 분석하고, 이를 보완하기 위한 부분적 나선 힘 궤적을 제안한다. 제안한 힘 궤적이 나선 힘 궤적에 비해 갖는 성능의 우수성을 증명하기 위하여 수치적 분석, 반복적 시뮬레이션, 그리고 로봇을 이용한 실험을 수행하였다.1 INTRODUCTION 1 1.1 Motivation: Peg-in-Hole Assembly 1 1.2 Contributions of Thesis 2 1.3 Overview of Thesis 3 2 COMPLIANCE BASED STRATEGY 5 2.1 Background & Related Works 5 2.2 Analysis of Peg-in-Hole Procedure 6 2.2.1 Contact Analysis 7 2.2.2 Basic Idea 9 2.3 Peg-in-Hole Strategy 12 2.3.1 Unit Motions 12 2.3.2 State of Strategy 13 2.3.3 Conditions for State Transition 15 2.4 Control Frameworks 18 2.4.1 Control for Compliant Behavior 18 2.4.2 Friction Compensate 20 2.4.3 Control Input for the Strategy 25 2.5 Experiment 29 2.5.1 Experiment Environment 29 2.5.2 Fixed Peg and Fixed Hole 31 2.5.2.1 Experiment Results 31 2.5.2.2 Analysis of Force and Control Gain 36 2.5.3 Peg-in-Hole with Multi Finger Hand 41 2.5.3.1 Object Grasping 42 2.5.3.2 Object In-Hand Manipulation 44 2.5.3.3 Experiment Results 49 2.5.4 With Upper Body Robot 50 2.5.4.1 Peg-in-Hole Procedure 52 2.5.4.2 Kinematics of Grasped Object 54 2.5.4.3 Control Frameworks 54 2.5.4.4 Experiment Results 56 2.6 Discussion 59 2.6.1 Peg-in-Hole Transition 59 2.6.2 Influential Issues 59 3 WRENCH TRAJECTORY 63 3.1 Problem Statement 64 3.1.1 Hole Search Process 64 3.1.2 Spiral Force Trajectory Analysis 66 3.2 Partial Spiral Force Trajectory 70 3.2.1 Force Trajectory with Tilted Posture 70 3.2.2 Probability to Three-point Contact 76 3.3 SIMULATION & EXPERIMENT 78 3.3.1 Simulation 78 3.3.2 Experiment 83 4 CONCLUSIONS 90 Abstract (In Korean) 102Docto

Analysis and Observations from the First Amazon Picking Challenge

This paper presents a overview of the inaugural Amazon Picking Challenge along with a summary of a survey conducted among the 26 participating teams. The challenge goal was to design an autonomous robot to pick items from a warehouse shelf. This task is currently performed by human workers, and there is hope that robots can someday help increase efficiency and throughput while lowering cost. We report on a 28-question survey posed to the teams to learn about each team's background, mechanism design, perception apparatus, planning and control approach. We identify trends in this data, correlate it with each team's success in the competition, and discuss observations and lessons learned based on survey results and the authors' personal experiences during the challenge

Learning to predict phases of manipulation tasks as hidden states

Phase transitions in manipulation tasks often occur when contacts between objects are made or broken. A switch of the phase can result in the robot’s actions suddenly influencing different aspects of its environment. Therefore, the boundaries between phases often correspond to constraints or subgoals of the manipulation task. In this paper, we investigate how the phases of manipulation tasks can be learned from data. The task is modeled as an autoregressive hidden Markov model, wherein the hidden phase transitions depend on the observed states. The model is learned from data using the expectation-maximization algorithm. We demonstrate the proposed method on both a pushing task and a pepper mill turning task. The proposed approach was compared to a standard autoregressive hidden Markov model. The experiments show that the learned models can accurately predict the transitions in phases during the manipulation tasks