Man-machine cooperation in advanced teleoperation

Teleoperation experiments at JPL have shown that advanced features in a telerobotic system are a necessary condition for good results, but that they are not sufficient to assure consistently good performance by the operators. Two or three operators are normally used during training and experiments to maintain the desired performance. An alternative to this multi-operator control station is a man-machine interface embedding computer programs that can perform some of the operator's functions. In this paper we present our first experiments with these concepts, in which we focused on the areas of real-time task monitoring and interactive path planning. In the first case, when performing a known task, the operator has an automatic aid for setting control parameters and camera views. In the second case, an interactive path planner will rank different path alternatives so that the operator will make the correct control decision. The monitoring function has been implemented with a neural network doing the real-time task segmentation. The interactive path planner was implemented for redundant manipulators to specify arm configurations across the desired path and satisfy geometric, task, and performance constraints

Performance evaluation of a six-axis generalized force-reflecting teleoperator

Work in real-time distributed computation and control has culminated in a prototype force-reflecting telemanipulation system having a dissimilar master (cable-driven, force-reflecting hand controller) and a slave (PUMA 560 robot with custom controller), an extremely high sampling rate (1000 Hz), and a low loop computation delay (5 msec). In a series of experiments with this system and five trained test operators covering over 100 hours of teleoperation, performance was measured in a series of generic and application-driven tasks with and without force feedback, and with control shared between teleoperation and local sensor referenced control. Measurements defining task performance included 100-Hz recording of six-axis force/torque information from the slave manipulator wrist, task completion time, and visual observation of predefined task errors. The task consisted of high precision peg-in-hole insertion, electrical connectors, velcro attach-de-attach, and a twist-lock multi-pin connector. Each task was repeated three times under several operating conditions: normal bilateral telemanipulation, forward position control without force feedback, and shared control. In shared control, orientation was locally servo controlled to comply with applied torques, while translation was under operator control. All performance measures improved as capability was added along a spectrum of capabilities ranging from pure position control through force-reflecting teleoperation and shared control. Performance was optimal for the bare-handed operator

불확실성을 포함하는 조립작업을 위한 컴플라이언스 기반 펙인홀 전략

학위논문 (박사) -- 서울대학교 대학원 : 융합과학기술대학원 융합과학부(지능형융합시스템전공), 2020. 8. 박재흥.The peg-in-hole assembly is a representative robotic task that involves physical contact with the external environment. The strategies generally involve performing the assembly task by estimating the contact state between the peg and the hole. The contact forces and moments, measured using force sensors, are primarily used to estimate the contact state. In this paper, in contrast to past research in the area, which has involved the utilization of such expensive devices as force/torque sensors or remote compliance mechanisms, an inexpensive method is proposed for peg-in-hole assembly without force feedback or passive compliance mechanisms. The method consists of an analysis of the state of contact between the peg and the hole as well as a strategy to overcome the inevitable positional uncertainty of the hole incurred in the recognition process. A control scheme was developed to yield compliant behavior from the robot with physical contact under the condition of hybrid position/force control. Proposed peg-in-hole strategy is based on compliance characteristics and generating the force and moment. The peg is inserted into the hole as it adapts to the external environment. The effectiveness of the proposed method was experimentally verified using a humanoid upper body robot with fifty degrees of freedom and a peg-in-hole apparatus with a small clearance (0.1 mm). Three cases of experiments were conducted; Assembling the peg attached to the arm in the hole fixed in the external environment, grasping a peg with an anthropomorphic hand and assembling it into a fixed hole, and grasping both peg and hole with both hands and assembling each other. In order to assemble the peg-in-hole through the proposed strategy by the humanoid upper body robot, I present a method of gripping an object, estimating the kinematics of the gripped object, and manipulating the gripped object. In addition to the cost aspect, which is the fundamental motivation for the proposed strategy, the experimental results show that the proposed strategy has advantages such as fast assembly time and high success rate, but has the disadvantage of unpredictable elapsed time. The reason for having a high variance value for the success time is that the spiral trajectory, which is most commonly used, is used. In this study, I analyze the efficiency of spiral force trajectory and propose an improved force trajectory. The proposed force trajectory reduces the distribution of elapsed time by eliminating the uncertainty in the time required to find a hole. The efficiency of the force trajectory is analyzed numerically, verified through repeated simulations, and verified by the actual experiment with humanoid upper body robot developed by Korea institute of industrial technology.펙인홀 조립은 로봇의 접촉 작업을 대표하는 작업으로, 펙인홀 조립 전략을 연구함으로써 산업 생산 분야의 조립작업에 적용할 수 있다. 펙인홀 조립작업은 일반적으로 펙과 홀 간의 접촉상태를 추정함으로써 이루어진다. 접촉상태를 추정하기 위해 가장 널리 쓰이는 방법은 힘 센서를 사용하는 것인데, 접촉 힘과 모멘트를 측정하여 접촉상태를 추정하는 방식이다. 만약 이러한 센서를 사용하지 않을 수 있다면, 하드웨어 비용과 소프트웨어 연산량 감소 등의 장점이 있음은 자명하다. 본 논문에서는 힘 센서 혹은 수동 컴플라이언스 장치를 사용하지 않는 펙인홀 전략을 제안한다. 홀에 대한 인식 오차 혹은 로봇의 제어 오차를 극복하기 위하여 먼저 펙과 홀의 접촉 가능 상태를 분석하고 로봇의 컴플라이언스 모션을 위한 제어 프레임워크를 디자인한다. 전략은 컴플라이언스 특징에 기반하며 펙에 힘과 모멘트를 생성시킴으로써 조립작업을 수행한다. 펙은 외부환경에 순응함으로써 홀에 삽입된다. 제안한 전략은 낮은 공차를 갖는 펙인홀 실험을 통해서 그 유효성이 검증된다. 펙과 홀을 로봇팔과 외부환경에 각각 고정된 환경에서의 실험, 인간형 로봇핸드를 이용하여 펙을 잡아서 고정된 홀에 삽입하는 실험, 그리고 테이블에 놓인 펙과 홀을 각각 로봇핸드로 파지하여 조립하는 총 세 가지의 실험을 수행하였다. 핸드로 펙을 파지하고 조작하기 위하여, 파지 방법과 핸드를 이용한 물체 조작 알고리즘을 간략히 소개하였다. 제안한 전략의 성능을 실험적으로 분석한 결과, 높은 조립 성공률을 갖는 대신 조립시간이 예측할 수 없는 단점이 나타나 이를 보완하기 위해서 렌치 궤적 또한 제안하였다. 먼저 가장 일반적으로 사용되는 나선 힘 궤적을 이용했을 때 조립 성공시간의 분산이 큰 이유를 확률개념을 이용해 분석하고, 이를 보완하기 위한 부분적 나선 힘 궤적을 제안한다. 제안한 힘 궤적이 나선 힘 궤적에 비해 갖는 성능의 우수성을 증명하기 위하여 수치적 분석, 반복적 시뮬레이션, 그리고 로봇을 이용한 실험을 수행하였다.1 INTRODUCTION 1 1.1 Motivation: Peg-in-Hole Assembly 1 1.2 Contributions of Thesis 2 1.3 Overview of Thesis 3 2 COMPLIANCE BASED STRATEGY 5 2.1 Background & Related Works 5 2.2 Analysis of Peg-in-Hole Procedure 6 2.2.1 Contact Analysis 7 2.2.2 Basic Idea 9 2.3 Peg-in-Hole Strategy 12 2.3.1 Unit Motions 12 2.3.2 State of Strategy 13 2.3.3 Conditions for State Transition 15 2.4 Control Frameworks 18 2.4.1 Control for Compliant Behavior 18 2.4.2 Friction Compensate 20 2.4.3 Control Input for the Strategy 25 2.5 Experiment 29 2.5.1 Experiment Environment 29 2.5.2 Fixed Peg and Fixed Hole 31 2.5.2.1 Experiment Results 31 2.5.2.2 Analysis of Force and Control Gain 36 2.5.3 Peg-in-Hole with Multi Finger Hand 41 2.5.3.1 Object Grasping 42 2.5.3.2 Object In-Hand Manipulation 44 2.5.3.3 Experiment Results 49 2.5.4 With Upper Body Robot 50 2.5.4.1 Peg-in-Hole Procedure 52 2.5.4.2 Kinematics of Grasped Object 54 2.5.4.3 Control Frameworks 54 2.5.4.4 Experiment Results 56 2.6 Discussion 59 2.6.1 Peg-in-Hole Transition 59 2.6.2 Influential Issues 59 3 WRENCH TRAJECTORY 63 3.1 Problem Statement 64 3.1.1 Hole Search Process 64 3.1.2 Spiral Force Trajectory Analysis 66 3.2 Partial Spiral Force Trajectory 70 3.2.1 Force Trajectory with Tilted Posture 70 3.2.2 Probability to Three-point Contact 76 3.3 SIMULATION & EXPERIMENT 78 3.3.1 Simulation 78 3.3.2 Experiment 83 4 CONCLUSIONS 90 Abstract (In Korean) 102Docto

Mastering Autonomous Assembly in Fusion Application with Learning-by-doing: a Peg-in-hole Study

Robotic peg-in-hole assembly is an essential task in robotic automation research. Reinforcement learning (RL) combined with deep neural networks (DNNs) lead to extraordinary achievements in this area. However, current RL-based approaches could hardly perform well under the unique environmental and mission requirements of fusion applications. Therefore, we have proposed a new designed RL-based method. Furthermore, unlike other approaches, we focus on innovations in the structure of DNNs instead of the RL model. Data from the RGB camera and force/torque (F/T) sensor as the input are fed into a multi-input branch network, and the best action in the current state is output by the network. All training and experiments are carried out in a realistic environment, and from the experiment result, this multi-sensor fusion approach has been shown to work well in rigid peg-in-hole assembly tasks with 0.1mm precision in uncertain and unstable environments

Robotic learning of force-based industrial manipulation tasks

Even with the rapid technological advancements, robots are still not the most comfortable machines to work with. Firstly, due to the separation of the robot and human workspace which imposes an additional financial burden. Secondly, due to the significant re-programming cost in case of changing products, especially in Small and Medium-sized Enterprises (SMEs). Therefore, there is a significant need to reduce the programming efforts required to enable robots to perform various tasks while sharing the same space with a human operator. Hence, the robot must be equipped with a cognitive and perceptual capabilities that facilitate human-robot interaction. Humans use their various senses to perform tasks such as vision, smell and taste. One sensethat plays a significant role in human activity is ’touch’ or ’force’. For example, holding a cup of tea, or making fine adjustments while inserting a key requires haptic information to achieve the task successfully. In all these examples, force and torque data are crucial for the successful completion of the activity. Also, this information implicitly conveys data about contact force, object stiffness, and many others. Hence, a deep understanding of the execution of such events can bridge the gap between humans and robots. This thesis is being directed to equip an industrial robot with the ability to deal with force perceptions and then learn force-based tasks using Learning from Demonstration (LfD).To learn force-based tasks using LfD, it is essential to extract task-relevant features from the force information. Then, knowledge must be extracted and encoded form the task-relevant features. Hence, the captured skills can be reproduced in a new scenario. In this thesis, these elements of LfD were achieved using different approaches based on the demonstrated task. In this thesis, four robotics problems were addressed using LfD framework. The first challenge was to filter out robots’ internal forces (irrelevant signals) using data-driven approach. The second robotics challenge was the recognition of the Contact State (CS) during assembly tasks. To tackle this challenge, a symbolic based approach was proposed, in which a force/torque signals; during demonstrated assembly, the task was encoded as a sequence of symbols. The third challenge was to learn a human-robot co-manipulation task based on LfD. In this case, an ensemble machine learning approach was proposed to capture such a skill. The last challenge in this thesis, was to learn an assembly task by demonstration with the presents of parts geometrical variation. Hence, a new learning approach based on Artificial Potential Field (APF) to learn a Peg-in-Hole (PiH) assembly task which includes no-contact and contact phases. To sum up, this thesis focuses on the use of data-driven approaches to learning force based task in an industrial context. Hence, different machine learning approaches were implemented, developed and evaluated in different scenarios. Then, the performance of these approaches was compared with mathematical modelling based approaches.</div

Evaluation of automated decisionmaking methodologies and development of an integrated robotic system simulation

A generic computer simulation for manipulator systems (ROBSIM) was implemented and the specific technologies necessary to increase the role of automation in various missions were developed. The specific items developed are: (1) capability for definition of a manipulator system consisting of multiple arms, load objects, and an environment; (2) capability for kinematic analysis, requirements analysis, and response simulation of manipulator motion; (3) postprocessing options such as graphic replay of simulated motion and manipulator parameter plotting; (4) investigation and simulation of various control methods including manual force/torque and active compliances control; (5) evaluation and implementation of three obstacle avoidance methods; (6) video simulation and edge detection; and (7) software simulation validation

Accomplishing task-invariant assembly strategies by means of an inherently accommodating robot arm

Despite the fact that the main advantage of robot manipulators was always meant to be their flexibility, they have not been applied widely to the assembly of industrial components in situations other than those where hard automation might be used. We identify the two main reasons for this as the 'fragility' of robot operation during tasks that involve contact, and the lack of an appropriate user interface. This thesis describes an attempt to address these problems.We survey the techniques that have been proposed to bring the performance of cur¬ rent industrial robot manipulators in line with expectations, and conclude that the main obstacle in realising a flexible assembly robot that exhibits robust and reliable behaviour is the problem of spatial uncertainty.Based on observations of the performance of position-controlled robot manipulators and what is involved during rigid-body part mating, we propose a model of assembly tasks that exploits the shape invariance of the part geometry across instances of a task. This allows us to escape from the problem of spatial uncertainty because we are 110 longer working in spatial terms. In addition, because the descriptions of assembly tasks that we derive are task-invariant, i.e. they are not dependent on part size or location, they lend themselves naturally to a task-level programming interface, thereby simplifying the process of programming an assembly robot.the process of programming an assembly robot. However, to test this approach empirically requires a manipulator that is able to control the force that it applies, as well as being sensitive to environmental constraints. The inertial properties of standard industrial manipulators preclude them from exhibiting this kind of behaviour. In order to solve this problem we designed and constructed a three degree of freedom, planar, direct-drive arm that is open-loop force-controllable (with respect to its end-point), and inherently accommodating during contact.In order to demonstrate the forgiving nature of operation of our robot arm we imple¬ mented a generic crank turning program that is independent of the geometry of the crank involved, i.e. no knowledge is required of the location or length of the crank. I11 order to demonstrate the viability of our proposed approach to assembly we pro¬ grammed our robot system to perform some representative tasks; the insertion of a peg into a hole, and the rotation of a block into a corner. These programs were tested on parts of various size and material, and in various locations in order to illustrate their invariant nature.We conclude that the problem of spatial uncertainty is in fact an artefact of the fact that current industrial manipulators are designed to be position controlled. The work described in this thesis shows that assembly robots, when appropriately designed, controlled and programmed, can be the reliable and flexible devices they were always meant to be