On the Value of Estimating Human Arm Stiffness during Virtual Teleoperation with Robotic Manipulators

Teleoperated robotic systems are widely spreading in multiple different fields, from hazardous environments exploration to surgery. In teleoperation, users directly manipulate a master device to achieve task execution at the slave robot side; this interaction is fundamental to guarantee both system stability and task execution performance. In this work, we propose a non-disruptive method to study the arm endpoint stiffness. We evaluate how users exploit the kinetic redundancy of the arm to achieve stability and precision during the execution of different tasks with different master devices. Four users were asked to perform two planar trajectories following virtual tasks using both a serial and a parallel link master device. Users' arm kinematics and muscular activation were acquired and combined with a user-specific musculoskeletal model to estimate the joint stiffness. Using the arm kinematic Jacobian, the arm end-point stiffness was derived. The proposed non-disruptive method is capable of estimating the arm endpoint stiffness during the execution of virtual teleoperated tasks. The obtained results are in accordance with the existing literature in human motor control and show, throughout the tested trajectory, a modulation of the arm endpoint stiffness that is affected by task characteristics and hand speed and acceleration

Development of a user experience enhanced teleoperation approach

In this paper, we have investigated various techniques that can be used to enhance user experience for robot teleoperation. In our teleoperation system design, the human operator are provided with both immersive visual feedback and intuitive skill transfer interface such that when controlling a telerobot arm, a user is able to feeļ in a first person perspective in terms of both visual and haptic sense. A number of high-tech devices including Omni haptic joystick, MYO armband, Oculus Rift DK2 headset, and Kinect v2 camera are integrated. The surface electromyography (sEMG) signal allows operator to naturally and efficiently transfer his/her motion skill to the robot, based on the properly designed elastic force feedback. For visual feedback, operators can control the pose of a camera on the head of the robot via the wearable visual headset, such that the operator is able to perceive from the roboţs perspective. Extensive tests have been performed with human subjects to evaluate the design, and the experimental results have shown that superior performance and better user experience have been achieved by the proposed method in comparison with the traditional methods

A quantitative taxonomy of human hand grasps

Background: A proper modeling of human grasping and of hand movements is fundamental for robotics, prosthetics, physiology and rehabilitation. The taxonomies of hand grasps that have been proposed in scientific literature so far are based on qualitative analyses of the movements and thus they are usually not quantitatively justified. Methods: This paper presents to the best of our knowledge the first quantitative taxonomy of hand grasps based on biomedical data measurements. The taxonomy is based on electromyography and kinematic data recorded from 40 healthy subjects performing 20 unique hand grasps. For each subject, a set of hierarchical trees are computed for several signal features. Afterwards, the trees are combined, first into modality-specific (i.e. muscular and kinematic) taxonomies of hand grasps and then into a general quantitative taxonomy of hand movements. The modality-specific taxonomies provide similar results despite describing different parameters of hand movements, one being muscular and the other kinematic. Results: The general taxonomy merges the kinematic and muscular description into a comprehensive hierarchical structure. The obtained results clarify what has been proposed in the literature so far and they partially confirm the qualitative parameters used to create previous taxonomies of hand grasps. According to the results, hand movements can be divided into five movement categories defined based on the overall grasp shape, finger positioning and muscular activation. Part of the results appears qualitatively in accordance with previous results describing kinematic hand grasping synergies. Conclusions: The taxonomy of hand grasps proposed in this paper clarifies with quantitative measurements what has been proposed in the field on a qualitative basis, thus having a potential impact on several scientific fields

MYOARM: prótesis robótica con sensado emg y entrenamiento con redes neuronales

El proyecto consiste en la creación de un brazo robótico controlado remotamente a través de un brazalete desarrollado por Thalmic Labs, el cual es capaz de leer los biopotenciales de los músculos residentes del muñón de los usuarios. Este proyecto, tiene como propósito la creación de una alternativa económica a las prótesis activas no invasivas que existen en la actualidad. Nuestra prótesis es capaz de realizar las mismas funciones, pero a un precio mucho más asequible. Para poder realizar todas las funciones que una articulación normal, el brazo cuenta con varios elementos: Cuerdas que simulan tendones y permiten el movimiento de los dedos, engranajes que permiten el giro de la muñeca y motores, los cuales son capaces de generar el movimiento en función de los datos extraídos del brazalete. El brazalete es el encargado de transmitir la información de la mano al brazo robótico a través de un módulo inalámbrico que lo conecta con el ordenador, donde la señal que extrae el brazalete pasa por un proceso de filtrado para quedarnos con la información que nos interesa y transmitirla mediante el puerto USB a un microcontrolador, el cual será el encargado de mover los motores según las señales que reciba. Para evitar errores en la medida de los sensores, la información recibida por el pc proveniente del brazalete pasa un proceso de entrenamiento mediante redes neuronales antes de ser enviada al brazo robótico.The project consists of the creation of a robotic arm controlled remotely through a brace developed by Thalmic Labs, which can read the biopotentials of the muscles of the limb of the users. This project aims to create an economic alternative to noninvasive active prostheses that exist today. Our prosthesis can perform the same functions but at a so much affordable price. To perform all the functions of a normal joint, the arm has several elements. Strings that simulate tendons and allow the movement of the fingers, gears that allow the rotation of the wrist and motors, which can generate movement based on the data extracted from the bracelet. The bracelet is responsible for transmitting information from the hand to the robotic arm through a wireless module that connects it with the computer, where the signal that extracts the bracelet goes through a filtering process to keep the information that interests us and Transmit it through the USB port to a microcontroller, which will be in charge of moving the engines according to the signals received. To avoid errors in the measurement of the sensors, the information received from the bracelet is trained in the computer using a Neural Network architecture before sending the information to the robotic arm.Universidad de Sevilla. TEP- 108: Robótica y Tecnología de Computadore

Formulation of a new gradient descent MARG orientation algorithm: case study on robot teleoperation

We introduce a novel magnetic angular rate gravity (MARG) sensor fusion algorithm for inertial measurement. The new algorithm improves the popular gradient descent (ʻMadgwick’) algorithm increasing accuracy and robustness while preserving computa- tional efficiency. Analytic and experimental results demonstrate faster convergence for multiple variations of the algorithm through changing magnetic inclination. Furthermore, decoupling of magnetic field variance from roll and pitch estimation is pro- ven for enhanced robustness. The algorithm is validated in a human-machine interface (HMI) case study. The case study involves hardware implementation for wearable robot teleoperation in both Virtual Reality (VR) and in real-time on a 14 degree-of-freedom (DoF) humanoid robot. The experiment fuses inertial (movement) and mechanomyography (MMG) muscle sensing to control robot arm movement and grasp simultaneously, demon- strating algorithm efficacy and capacity to interface with other physiological sensors. To our knowledge, this is the first such formulation and the first fusion of inertial measure- ment and MMG in HMI. We believe the new algorithm holds the potential to impact a very wide range of inertial measurement applications where full orientation necessary. Physiological sensor synthesis and hardware interface further provides a foundation for robotic teleoperation systems with necessary robustness for use in the field

Safe Haptics-enabled Patient-Robot Interaction for Robotic and Telerobotic Rehabilitation of Neuromuscular Disorders: Control Design and Analysis

Motivation: Current statistics show that the population of seniors and the incidence rate of age-related neuromuscular disorders are rapidly increasing worldwide. Improving medical care is likely to increase the survival rate but will result in even more patients in need of Assistive, Rehabilitation and Assessment (ARA) services for extended periods which will place a significant burden on the world\u27s healthcare systems. In many cases, the only alternative is limited and often delayed outpatient therapy. The situation will be worse for patients in remote areas. One potential solution is to develop technologies that provide efficient and safe means of in-hospital and in-home kinesthetic rehabilitation. In this regard, Haptics-enabled Interactive Robotic Neurorehabilitation (HIRN) systems have been developed. Existing Challenges: Although there are specific advantages with the use of HIRN technologies, there still exist several technical and control challenges, e.g., (a) absence of direct interactive physical interaction between therapists and patients; (b) questionable adaptability and flexibility considering the sensorimotor needs of patients; (c) limited accessibility in remote areas; and (d) guaranteeing patient-robot interaction safety while maximizing system transparency, especially when high control effort is needed for severely disabled patients, when the robot is to be used in a patient\u27s home or when the patient experiences involuntary movements. These challenges have provided the motivation for this research. Research Statement: In this project, a novel haptics-enabled telerobotic rehabilitation framework is designed, analyzed and implemented that can be used as a new paradigm for delivering motor therapy which gives therapists direct kinesthetic supervision over the robotic rehabilitation procedure. The system also allows for kinesthetic remote and ultimately in-home rehabilitation. To guarantee interaction safety while maximizing the performance of the system, a new framework for designing stabilizing controllers is developed initially based on small-gain theory and then completed using strong passivity theory. The proposed control framework takes into account knowledge about the variable biomechanical capabilities of the patient\u27s limb(s) in absorbing interaction forces and mechanical energy. The technique is generalized for use for classical rehabilitation robotic systems to realize patient-robot interaction safety while enhancing performance. In the next step, the proposed telerobotic system is studied as a modality of training for classical HIRN systems. The goal is to first model and then regenerate the prescribed kinesthetic supervision of an expert therapist. To broaden the population of patients who can use the technology and HIRN systems, a new control strategy is designed for patients experiencing involuntary movements. As the last step, the outcomes of the proposed theoretical and technological developments are translated to designing assistive mechatronic tools for patients with force and motion control deficits. This study shows that proper augmentation of haptic inputs can not only enhance the transparency and safety of robotic and telerobotic rehabilitation systems, but it can also assist patients with force and motion control deficiencies

Electromyography control of robotic systems

This work describes research of myoelectric interfaces and their application for controlling robotic systems. Hand gesture data collection software has been created. The neural network was designed and trained to recognize various gestures. The accuracy was 0.96 for four gestures and 0.925 for seven gestures. The prototype of myoelectric signals controlled robot with two degrees of freedom was created. Wireless direct control via bluetooth was implemented.This work describes research of myoelectric interfaces and their application for controlling robotic systems. Hand gesture data collection software has been created. The neural network was designed and trained to recognize various gestures. The accuracy was 0.96 for four gestures and 0.925 for seven gestures. The prototype of myoelectric signals controlled robot with two degrees of freedom was created. Wireless direct control via bluetooth was implemented

Teleoperation control of Baxter robot using Kalman filter-based sensor fusion

Kalman filter has been successfully applied to fuse the motion capture data collected from Kinect sensor and a pair of MYO armbands to teleoperate a robot. A new strategy utilizing the vector approach has been developed to accomplish a specific motion capture task. The arm motion of the operator is captured by a Kinect sensor and programmed with Processing software. Two MYO armbands with the inertial measurement unit embedded are worn on the operator's arm, which is used to detect the upper arm motion of the human operator. This is utilized to recognize and to calculate the precise speed of the physical motion of the operator's arm. User Datagram Protocol is employed to send the human movement to a simulated Baxter robot arm for teleoperation. In order to obtain joint angles for human limb utilizing vector approach, RosPy and Python script programming has been utilized. A series of experiments have been conducted to test the performance of the proposed technique, which provides the basis for the teleoperation of simulated Baxter robot

User Experience Enchanced Interface ad Controller Design for Human-Robot Interaction

The robotic technologies have been well developed recently in various fields, such as medical services, industrial manufacture and aerospace. Despite their rapid development, how to deal with the uncertain envi-ronment during human-robot interactions effectively still remains un-resolved. The current artificial intelligence (AI) technology does not support robots to fulfil complex tasks without human’s guidance. Thus, teleoperation, which means remotely controlling a robot by a human op-erator, is indispensable in many scenarios. It is an important and useful tool in research fields. This thesis focuses on the study of designing a user experience (UX) enhanced robot controller, and human-robot in-teraction interfaces that try providing human operators an immersion perception of teleoperation. Several works have been done to achieve the goal.First, to control a telerobot smoothly, a customised variable gain con-trol method is proposed where the stiffness of the telerobot varies with the muscle activation level extracted from signals collected by the surface electromyograph(sEMG) devices. Second, two main works are conducted to improve the user-friendliness of the interaction interfaces. One is that force feedback is incorporated into the framework providing operators with haptic feedback to remotely manipulate target objects. Given the high cost of force sensor, in this part of work, a haptic force estimation algorithm is proposed where force sensor is no longer needed. The other main work is developing a visual servo control system, where a stereo camera is mounted on the head of a dual arm robots offering operators real-time working situations. In order to compensate the internal and ex-ternal uncertainties and accurately track the stereo camera’s view angles along planned trajectories, a deterministic learning techniques is utilised, which enables reusing the learnt knowledge before current dynamics changes and thus features increasing the learning efficiency. Third, in-stead of sending commands to the telerobts by joy-sticks, keyboards or demonstrations, the telerobts are controlled directly by the upper limb motion of the human operator in this thesis. Algorithm that utilised the motion signals from inertial measurement unit (IMU) sensor to captures humans’ upper limb motion is designed. The skeleton of the operator is detected by Kinect V2 and then transformed and mapped into the joint positions of the controlled robot arm. In this way, the upper limb mo-tion signals from the operator is able to act as reference trajectories to the telerobts. A more superior neural networks (NN) based trajectory controller is also designed to track the generated reference trajectory. Fourth, to further enhance the human immersion perception of teleop-eration, the virtual reality (VR) technique is incorporated such that the operator can make interaction and adjustment of robots easier and more accurate from a robot’s perspective.Comparative experiments have been performed to demonstrate the effectiveness of the proposed design scheme. Tests with human subjects were also carried out for evaluating the interface design