Characterizing the correlation between motor cortical neural firing and grasping kinematics

The hand has evolved to allow specialized interactions with our surroundings that define much of what makes us human. Comprised of numerous joints allowing 23 separate degrees-of-freedom (DoF) (joint motions) of movement, the hand and wrist are exceedingly complex. In order to better understand the constraints and principles underlying the neural control of the hand, we have carried out a series of neurophysiological experiments with monkeys performing a variety of reaching and grasping tasks. This work uses linear regression and low dimensional analysis to probe the neural representation of hand kinematics.We find that the kinematics of the three wrist DoFs (flexion, abduction and rotation) are rashly independent from hand-shape DoFs, and are considered separately. With respect to the wrist DoF, we show that the firing patterns of individual motor cortical cells are more linearly related to joint position than joint angular velocity. Using tuning functions from multivariate linear regressions, the firing rates from a population of cells accurately predicted three DoF of wrist orientation. We used principal components analysis to simplify the complex kinematics of the hand. Although the majority of the variability in hand kinematics can be explained with a small number (~7) of characteristic hand shapes (synergies), we find that these synergies do not capture the majority of neural variability. Both higher-order and lower-order synergies are well represented in the neural data. Although the kinematic synergies do not fully characterize neural firing, they can be utilized to simplify hand shape decoding. Using an optimal linear estimator, we predicted the average wrist and hand shape from the firing rates of 327 motor cortical cells with an accuracy as high as 92%. Individual motor cortical neurons are not well correlated with single joint variables; rather, they correlate with a number of joints in a complex way. This work provides evidence that hand movements are likely controlled through an intricate network of motor systems, of which motor cortical neurons contribute by making fine adjustments to a basic substrate. Further understanding of the control system will be gained by establishing a model that captures both the hand kinematic and neural variability

A Continuous Grasp Representation for the Imitation Learning of Grasps on Humanoid Robots

Models and methods are presented which enable a humanoid robot to learn reusable, adaptive grasping skills. Mechanisms and principles in human grasp behavior are studied. The findings are used to develop a grasp representation capable of retaining specific motion characteristics and of adapting to different objects and tasks. Based on the representation a framework is proposed which enables the robot to observe human grasping, learn grasp representations, and infer executable grasping actions

불확실성을 포함하는 조립작업을 위한 컴플라이언스 기반 펙인홀 전략

학위논문 (박사) -- 서울대학교 대학원 : 융합과학기술대학원 융합과학부(지능형융합시스템전공), 2020. 8. 박재흥.The peg-in-hole assembly is a representative robotic task that involves physical contact with the external environment. The strategies generally involve performing the assembly task by estimating the contact state between the peg and the hole. The contact forces and moments, measured using force sensors, are primarily used to estimate the contact state. In this paper, in contrast to past research in the area, which has involved the utilization of such expensive devices as force/torque sensors or remote compliance mechanisms, an inexpensive method is proposed for peg-in-hole assembly without force feedback or passive compliance mechanisms. The method consists of an analysis of the state of contact between the peg and the hole as well as a strategy to overcome the inevitable positional uncertainty of the hole incurred in the recognition process. A control scheme was developed to yield compliant behavior from the robot with physical contact under the condition of hybrid position/force control. Proposed peg-in-hole strategy is based on compliance characteristics and generating the force and moment. The peg is inserted into the hole as it adapts to the external environment. The effectiveness of the proposed method was experimentally verified using a humanoid upper body robot with fifty degrees of freedom and a peg-in-hole apparatus with a small clearance (0.1 mm). Three cases of experiments were conducted; Assembling the peg attached to the arm in the hole fixed in the external environment, grasping a peg with an anthropomorphic hand and assembling it into a fixed hole, and grasping both peg and hole with both hands and assembling each other. In order to assemble the peg-in-hole through the proposed strategy by the humanoid upper body robot, I present a method of gripping an object, estimating the kinematics of the gripped object, and manipulating the gripped object. In addition to the cost aspect, which is the fundamental motivation for the proposed strategy, the experimental results show that the proposed strategy has advantages such as fast assembly time and high success rate, but has the disadvantage of unpredictable elapsed time. The reason for having a high variance value for the success time is that the spiral trajectory, which is most commonly used, is used. In this study, I analyze the efficiency of spiral force trajectory and propose an improved force trajectory. The proposed force trajectory reduces the distribution of elapsed time by eliminating the uncertainty in the time required to find a hole. The efficiency of the force trajectory is analyzed numerically, verified through repeated simulations, and verified by the actual experiment with humanoid upper body robot developed by Korea institute of industrial technology.펙인홀 조립은 로봇의 접촉 작업을 대표하는 작업으로, 펙인홀 조립 전략을 연구함으로써 산업 생산 분야의 조립작업에 적용할 수 있다. 펙인홀 조립작업은 일반적으로 펙과 홀 간의 접촉상태를 추정함으로써 이루어진다. 접촉상태를 추정하기 위해 가장 널리 쓰이는 방법은 힘 센서를 사용하는 것인데, 접촉 힘과 모멘트를 측정하여 접촉상태를 추정하는 방식이다. 만약 이러한 센서를 사용하지 않을 수 있다면, 하드웨어 비용과 소프트웨어 연산량 감소 등의 장점이 있음은 자명하다. 본 논문에서는 힘 센서 혹은 수동 컴플라이언스 장치를 사용하지 않는 펙인홀 전략을 제안한다. 홀에 대한 인식 오차 혹은 로봇의 제어 오차를 극복하기 위하여 먼저 펙과 홀의 접촉 가능 상태를 분석하고 로봇의 컴플라이언스 모션을 위한 제어 프레임워크를 디자인한다. 전략은 컴플라이언스 특징에 기반하며 펙에 힘과 모멘트를 생성시킴으로써 조립작업을 수행한다. 펙은 외부환경에 순응함으로써 홀에 삽입된다. 제안한 전략은 낮은 공차를 갖는 펙인홀 실험을 통해서 그 유효성이 검증된다. 펙과 홀을 로봇팔과 외부환경에 각각 고정된 환경에서의 실험, 인간형 로봇핸드를 이용하여 펙을 잡아서 고정된 홀에 삽입하는 실험, 그리고 테이블에 놓인 펙과 홀을 각각 로봇핸드로 파지하여 조립하는 총 세 가지의 실험을 수행하였다. 핸드로 펙을 파지하고 조작하기 위하여, 파지 방법과 핸드를 이용한 물체 조작 알고리즘을 간략히 소개하였다. 제안한 전략의 성능을 실험적으로 분석한 결과, 높은 조립 성공률을 갖는 대신 조립시간이 예측할 수 없는 단점이 나타나 이를 보완하기 위해서 렌치 궤적 또한 제안하였다. 먼저 가장 일반적으로 사용되는 나선 힘 궤적을 이용했을 때 조립 성공시간의 분산이 큰 이유를 확률개념을 이용해 분석하고, 이를 보완하기 위한 부분적 나선 힘 궤적을 제안한다. 제안한 힘 궤적이 나선 힘 궤적에 비해 갖는 성능의 우수성을 증명하기 위하여 수치적 분석, 반복적 시뮬레이션, 그리고 로봇을 이용한 실험을 수행하였다.1 INTRODUCTION 1 1.1 Motivation: Peg-in-Hole Assembly 1 1.2 Contributions of Thesis 2 1.3 Overview of Thesis 3 2 COMPLIANCE BASED STRATEGY 5 2.1 Background & Related Works 5 2.2 Analysis of Peg-in-Hole Procedure 6 2.2.1 Contact Analysis 7 2.2.2 Basic Idea 9 2.3 Peg-in-Hole Strategy 12 2.3.1 Unit Motions 12 2.3.2 State of Strategy 13 2.3.3 Conditions for State Transition 15 2.4 Control Frameworks 18 2.4.1 Control for Compliant Behavior 18 2.4.2 Friction Compensate 20 2.4.3 Control Input for the Strategy 25 2.5 Experiment 29 2.5.1 Experiment Environment 29 2.5.2 Fixed Peg and Fixed Hole 31 2.5.2.1 Experiment Results 31 2.5.2.2 Analysis of Force and Control Gain 36 2.5.3 Peg-in-Hole with Multi Finger Hand 41 2.5.3.1 Object Grasping 42 2.5.3.2 Object In-Hand Manipulation 44 2.5.3.3 Experiment Results 49 2.5.4 With Upper Body Robot 50 2.5.4.1 Peg-in-Hole Procedure 52 2.5.4.2 Kinematics of Grasped Object 54 2.5.4.3 Control Frameworks 54 2.5.4.4 Experiment Results 56 2.6 Discussion 59 2.6.1 Peg-in-Hole Transition 59 2.6.2 Influential Issues 59 3 WRENCH TRAJECTORY 63 3.1 Problem Statement 64 3.1.1 Hole Search Process 64 3.1.2 Spiral Force Trajectory Analysis 66 3.2 Partial Spiral Force Trajectory 70 3.2.1 Force Trajectory with Tilted Posture 70 3.2.2 Probability to Three-point Contact 76 3.3 SIMULATION & EXPERIMENT 78 3.3.1 Simulation 78 3.3.2 Experiment 83 4 CONCLUSIONS 90 Abstract (In Korean) 102Docto

Learning to grasp and extract affordances: the Integrated Learning of Grasps and Affordances (ILGA) model

The activity of certain parietal neurons has been interpreted as encoding affordances (directly perceivable opportunities) for grasping. Separate computational models have been developed for infant grasp learning and affordance learning, but no single model has yet combined these processes in a neurobiologically plausible way. We present the Integrated Learning of Grasps and Affordances (ILGA) model that simultaneously learns grasp affordances from visual object features and motor parameters for planning grasps using trial-and-error reinforcement learning. As in the Infant Learning to Grasp Model, we model a stage of infant development prior to the onset of sophisticated visual processing of hand–object relations, but we assume that certain premotor neurons activate neural populations in primary motor cortex that synergistically control different combinations of fingers. The ILGA model is able to extract affordance representations from visual object features, learn motor parameters for generating stable grasps, and generalize its learned representations to novel objects

Coupled dynamical system based hand-arm grasp planning under real-time perturbations

Robustness to perturbation has been advocated as a key element to robot control and efforts in that direction are numerous. While in essence these approaches aim at "endowing robots with a flexibility similar to that displayed by humans", few have actually looked at how humans react in the face of fast perturbations. We recorded the kinematic data from human subjects during grasping motions under very fast perturbations. Results show a strong coupling between the reach and grasp components of the task that enables rapid adaptation of the fingers in coordination with the hand posture when the target object is perturbed. We develop a robot controller based on Coupled Dynamical Systems that exploits coupling between two dynamical systems driving the hand and finger motions. This offers a compact encoding for a variety of reach and grasp motions that adapts on-the-fly to perturbations without the need for any re-planning. To validate the model we control the motion of the iCub robot when reaching for different objects

Coupled dynamical system based arm-hand grasping model for learning fast adaptation strategies

Performing manipulation tasks interactively in real environments requires a high degree of accuracy and stability. At the same time, when one cannot assume a fully deterministic and static environment, one must endow the robot with the ability to react rapidly to sudden changes in the environment. These considerations make the task of reach and grasp difficult to deal with. We follow a programming by demonstration (PbD) approach to the problem and take inspiration from the way humans adapt their reach and grasp motions when perturbed. This is in sharp contrast to previous work in PbD that uses unperturbed motions for training the system and then applies perturbation solely during the testing phase. In this work, we record the kinematics of arm and fingers of human subjects during unperturbed and perturbed reach and grasp motions. In the perturbed demonstrations, the target’s location is changed suddenly after the onset of the motion. Data show a strong coupling between the hand transport and finger motions. We hypothesize that this coupling enables the subject to seamlessly and rapidly adapt the finger motion in coordination with the hand posture. To endow our robot with this competence, we develop a Coupled Dynamical System based controller, whereby two dynamical systems driving the hand and finger motions are coupled. This offers a compact encoding for reach-to-grasp motions that ensures fast adaptation with zero latency for re-planning. We show in simulation and on the real iCub robot that this coupling ensures smooth and “human-like” motions. We demonstrate the performance of our model under spatial, temporal and grasp type perturbations which show that reaching the target with coordinated hand-arm motion is necessary for the success of the task

Sensitive Manipulation

PhD thesisThis thesis presents an effective alternative to the traditionalapproach to robotic manipulation. In our approach, manipulation ismainly guided by tactile feedback as opposed to vision. Themotivation comes from the fact that manipulating an object impliescoming in contact with it, consequently, directly sensing physicalcontact seems more important than vision to control theinteraction of the object and the robot. In this work, thetraditional approach of a highly precise arm and vision systemcontrolled by a model-based architecture is replaced by one thatuses a low mechanical impedance arm with dense tactile sensing andexploration capabilities run by a behavior-based architecture.The robot OBRERO has been built to implement this approach. Newtactile sensing technology has been developed and mounted on therobot's hand. These sensors are biologically inspired and presentmore adequate features for manipulation than those of state of theart tactile sensors. The robot's limb was built with compliantactuators, which present low mechanical impedance, to make theinteraction between the robot and the environment safer than thatof a traditional high-stiffness arm. A new actuator was created tofit in the hand size constraints. The reduced precision ofOBRERO's limb is compensated by the capability of explorationgiven by the tactile sensors, actuators and the softwarearchitecture.The success of this approach is shown by picking up objects in anunmodelled environment. This task, simple for humans, has been achallenge for robots. The robot can deal with new, unmodelledobjects. OBRERO can come gently in contact, explore, lift, andplace the object in a different location. It can also detectslippage and external forces acting on an object while it is held.Each one of these steps are done by using tactile feedback. Thistask can be done with very light objects with no fixtures and onslippery surfaces

Sensitive manipulation

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2007.This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.Includes bibliographical references (p. 161-172).This thesis presents an effective alternative to the traditional approach to robotic manipulation. In our approach, manipulation is mainly guided by tactile feedback as opposed to vision. The motivation comes from the fact that manipulating an object implies coming in contact with it, consequently, directly sensing physical contact seems more important than vision to control the interaction of the object and the robot. In this work, the traditional approach of a highly precise arm and vision system controlled by a model-based architecture is replaced by one that uses a low mechanical impedance arm with dense tactile sensing and exploration capabilities run by a behavior-based architecture. The robot OBRERO has been built to implement this approach. New tactile sensing technology has been developed and mounted on the robot's hand. These sensors are biologically inspired and present more adequate features for manipulation than those of state of the art tactile sensors. The robot's limb was built with compliant actuators, which present low mechanical impedance, to make the interaction between the robot and the environment safer than that of a traditional high-stiffness arm. A new actuator was created to fit in the hand size constraints.(cont.) The reduced precision of OBRERO's limb is compensated by the capability of exploration given by the tactile sensors, actuators and the software architecture. The success of this approach is shown by picking up objects in an unmodelled environment. This task, simple for humans, has been a challenge for robots. The robot can deal with new, unmodelled objects. OBRERO can come gently in contact, explore, lift, and place the object in a different location. It can also detect slippage and external forces acting on an object while it is held. Each one of these steps are done by using tactile feedback. This task can be done with very light objects with no fixtures and on slippery surfaces.by Eduardo Rafael Torres Jara.Ph.D

Adaptive and reconfigurable robotic gripper hands with a meso-scale gripping range

Grippers and robotic hands are essential and important end-effectors of robotic manipulators. Developing a gripper hand that can grasp a large variety of objects precisely and stably is still an aspiration even though research in this area has been carried out for several decades. This thesis provides a development approach and a series of gripper hands which can bridge the gap between micro-gripper and macro-gripper by extending the gripping range to the mesoscopic scale (meso-scale). Reconfigurable topology and variable mobility of the design offer versatility and adaptability for the changing environment and demands. By investigating human grasping behaviours and the unique structures of human hand, a CFB-based finger joint for anthropomorphic finger is developed to mimic a human finger with a large grasping range. The centrodes of CFB mechanism are explored and a contact-aided CFB mechanism is developed to increase stiffness of finger joints. An integrated gripper structure comprising cross four-bar (CFB) and remote-centre-of-motion (RCM) mechanisms is developed to mimic key functionalities of human hand. Kinematics and kinetostatic analyses of the CFB mechanism for multimode gripping are conducted to achieve passive-adjusting motion. A novel RCM-based finger with angular, parallel and underactuated motion is invented. Kinematics and stable gripping analyses of the RCM-based multi-motion finger are also investigated. The integrated design with CFB and RCM mechanisms provides a novel concept of a multi-mode gripper that aims to tackle the challenge of changing over for various sizes of objects gripping in mesoscopic scale range. Based on the novel designed mechanisms and design philosophy, a class of gripper hands in terms of adaptive meso-grippers, power-precision grippers and reconfigurable hands are developed. The novel features of the gripper hands are one degree of freedom (DoF), self-adaptive, reconfigurable and multi-mode. Prototypes are manufactured by 3D printing and the grasping abilities are tested to verify the design approach.EPSR