Teaching humanoid robotics by means of human teleoperation through RGB-D sensors

This paper presents a graduate course project on humanoid robotics offered by the University of Padova. The target is to safely lift an object by teleoperating a small humanoid. Students have to map human limbs into robot joints, guarantee the robot stability during the motion, and teleoperate the robot to perform the correct movement. We introduce the following innovative aspects with respect to classical robotic classes: i) the use of humanoid robots as teaching tools; ii) the simplification of the stable locomotion problem by exploiting the potential of teleoperation; iii) the adoption of a Project-Based Learning constructivist approach as teaching methodology. The learning objectives of both course and project are introduced and compared with the students\u2019 background. Design and constraints students have to deal with are reported, together with the amount of time they and their instructors dedicated to solve tasks. A set of evaluation results are provided in order to validate the authors\u2019 purpose, including the students\u2019 personal feedback. A discussion about possible future improvements is reported, hoping to encourage further spread of educational robotics in schools at all levels

Multi-robot cooperative platform : a task-oriented teleoperation paradigm

This thesis proposes the study and development of a teleoperation system based on multi-robot cooperation under the task oriented teleoperation paradigm: Multi-Robot Cooperative Paradigm, MRCP. In standard teleoperation, the operator uses the master devices to control the remote slave robot arms. These arms reproduce the desired movements and perform the task. With the developed work, the operator can virtually manipulate an object. MRCP automatically generates the arms orders to perform the task. The operator does not have to solve situations arising from possible restrictions that the slave arms may have. The research carried out is therefore aimed at improving the accuracy teleoperation tasks in complex environments, particularly in the field of robot assisted minimally invasive surgery. This field requires patient safety and the workspace entails many restrictions to teleoperation. MRCP can be defined as a platform composed of several robots that cooperate automatically to perform a teleoperated task, creating a robotic system with increased capacity (workspace volume, accessibility, dexterity ...). The cooperation is based on transferring the task between robots when necessary to enable a smooth task execution. The MRCP control evaluates the suitability of each robot to continue with the ongoing task and the optimal time to execute a task transfer between the current selected robot and the best candidate to continue with the task. From the operator¿s point of view, MRCP provides an interface that enables the teleoperation though the task-oriented paradigm: operator orders are translated into task actions instead of robot orders. This thesis is structured as follows: The first part is dedicated to review the current solutions in the teleoperation of complex tasks and compare them with those proposed in this research. The second part of the thesis presents and reviews in depth the different evaluation criteria to determine the suitability of each robot to continue with the execution of a task, considering the configuration of the robots and emphasizing the criterion of dexterity and manipulability. The study reviews the different required control algorithms to enable the task oriented telemanipulation. This proposed teleoperation paradigm is transparent to the operator. Then, the Thesis presents and analyses several experimental results using MRCP in the field of minimally invasive surgery. These experiments study the effectiveness of MRCP in various tasks requiring the cooperation of two hands. A type task is used: a suture using minimally invasive surgery technique. The analysis is done in terms of execution time, economy of movement, quality and patient safety (potential damage produced by undesired interaction between the tools and the vital tissues of the patient). The final part of the thesis proposes the implementation of different virtual aids and restrictions (guided teleoperation based on haptic visual and audio feedback, protection of restricted workspace regions, etc.) using the task oriented teleoperation paradigm. A framework is defined for implementing and applying a basic set of virtual aids and constraints within the framework of a virtual simulator for laparoscopic abdominal surgery. The set of experiments have allowed to validate the developed work. The study revealed the influence of virtual aids in the learning process of laparoscopic techniques. It has also demonstrated the improvement of learning curves, which paves the way for its implementation as a methodology for training new surgeons.Aquesta tesi doctoral proposa l'estudi i desenvolupament d'un sistema de teleoperació basat en la cooperació multi-robot sota el paradigma de la teleoperació orientada a tasca: Multi-Robot Cooperative Paradigm, MRCP. En la teleoperació clàssica, l'operador utilitza els telecomandaments perquè els braços robots reprodueixin els seus moviments i es realitzi la tasca desitjada. Amb el treball realitzat, l'operador pot manipular virtualment un objecte i és mitjançant el MRCP que s'adjudica a cada braç les ordres necessàries per realitzar la tasca, sense que l'operador hagi de resoldre les situacions derivades de possibles restriccions que puguin tenir els braços executors. La recerca desenvolupada està doncs orientada a millorar la teleoperació en tasques de precisió en entorns complexos i, en particular, en el camp de la cirurgia mínimament invasiva assistida per robots. Aquest camp imposa condicions de seguretat del pacient i l'espai de treball comporta moltes restriccions a la teleoperació. MRCP es pot definir com a una plataforma formada per diversos robots que cooperen de forma automàtica per dur a terme una tasca teleoperada, generant un sistema robòtic amb capacitats augmentades (volums de treball, accessibilitat, destresa,...). La cooperació es basa en transferir la tasca entre robots a partir de determinar quin és aquell que és més adequat per continuar amb la seva execució i el moment òptim per realitzar la transferència de la tasca entre el robot actiu i el millor candidat a continuar-la. Des del punt de vista de l'operari, MRCP ofereix una interfície de teleoperació que permet la realització de la teleoperació mitjançant el paradigma d'ordres orientades a la tasca: les ordres es tradueixen en accions sobre la tasca en comptes d'estar dirigides als robots. Aquesta tesi està estructurada de la següent manera: Primerament es fa una revisió de l'estat actual de les diverses solucions desenvolupades actualment en el camp de la teleoperació de tasques complexes, comparant-les amb les proposades en aquest treball de recerca. En el segon bloc de la tesi es presenten i s'analitzen a fons els diversos criteris per determinar la capacitat de cada robot per continuar l'execució d'una tasca, segons la configuració del conjunt de robots i fent especial èmfasi en el criteri de destresa i manipulabilitat. Seguint aquest estudi, es presenten els diferents processos de control emprats per tal d'assolir la telemanipulació orientada a tasca de forma transparent a l'operari. Seguidament es presenten diversos resultats experimentals aplicant MRCP al camp de la cirurgia mínimament invasiva. En aquests experiments s'estudia l'eficàcia de MRCP en diverses tasques que requereixen de la cooperació de dues mans. S'ha escollit una tasca tipus: sutura amb tècnica de cirurgia mínimament invasiva. L'anàlisi es fa en termes de temps d'execució, economia de moviment, qualitat i seguretat del pacient (potencials danys causats per la interacció no desitjada entre les eines i els teixits vitals del pacient). Finalment s'ha estudiat l'ús de diferents ajudes i restriccions virtuals (guiat de la teleoperació via retorn hàptic, visual o auditiu, protecció de regions de l'espai de treball, etc) dins el paradigma de teleoperació orientada a tasca. S'ha definint un marc d'aplicació base i implementant un conjunt de restriccions virtuals dins el marc d'un simulador de cirurgia laparoscòpia abdominal. El conjunt d'experiments realitzats han permès validar el treball realitzat. Aquest estudi ha permès determinar la influencia de les ajudes virtuals en el procés d'aprenentatge de les tècniques laparoscòpiques. S'ha evidenciat una millora en les corbes d'aprenentatge i obre el camí a la seva implantació com a metodologia d'entrenament de nous cirurgians.Postprint (published version

An 8-DOF dual-arm system for advanced teleoperation performance experiments

This paper describes the electro-mechanical and control features of an 8-DOF manipulator manufactured by AAI Corporation and installed at the Jet Propulsion Lab. (JPL) in a dual-arm setting. The 8-DOF arm incorporates a variety of features not found in other lab or industrial manipulators. Some of the unique features are: 8-DOF revolute configuration with no lateral offsets at joint axes; 1 to 5 payload to weight ratio with 20 kg (44 lb) payload at a 1.75 m (68.5 in.) reach; joint position measurement with dual relative encoders and potentiometer; infinite roll of joint 8 with electrical and fiber optic slip rings; internal fiber optic link of 'smart' end effectors; four-axis wrist; graphite epoxy links; high link and joint stiffness; use of an upgraded JPL Universal Motor Controller (UMC) capable of driving up to 16 joints. The 8-DOF arm is equipped with a 'smart' end effector which incorporates a 6-DOF forcemoment sensor at the end effector base and grasp force sensors at the base of the parallel jaws. The 8-DOF arm is interfaced to a 6 DOF force reflecting hand controller. The same system is duplicated for and installed at NASA-Langley

Risk-aware Path and Motion Planning for a Tethered Aerial Visual Assistant in Unstructured or Confined Environments

This research aims at developing path and motion planning algorithms for a tethered Unmanned Aerial Vehicle (UAV) to visually assist a teleoperated primary robot in unstructured or confined environments. The emerging state of the practice for nuclear operations, bomb squad, disaster robots, and other domains with novel tasks or highly occluded environments is to use two robots, a primary and a secondary that acts as a visual assistant to overcome the perceptual limitations of the sensors by providing an external viewpoint. However, the benefits of using an assistant have been limited for at least three reasons: (1) users tend to choose suboptimal viewpoints, (2) only ground robot assistants are considered, ignoring the rapid evolution of small unmanned aerial systems for indoor flying, (3) introducing a whole crew for the second teleoperated robot is not cost effective, may introduce further teamwork demands, and therefore could lead to miscommunication. This dissertation proposes to use an autonomous tethered aerial visual assistant to replace the secondary robot and its operating crew. Along with a pre-established theory of viewpoint quality based on affordances, this dissertation aims at defining and representing robot motion risk in unstructured or confined environments. Based on those theories, a novel high level path planning algorithm is developed to enable risk-aware planning, which balances the tradeoff between viewpoint quality and motion risk in order to provide safe and trustworthy visual assistance flight. The planned flight trajectory is then realized on a tethered UAV platform. The perception and actuation are tailored to fit the tethered agent in the form of a low level motion suite, including a novel tether-based localization model with negligible computational overhead, motion primitives for the tethered airframe based on position and velocity control, and two differentComment: Ph.D Dissertatio

Risk-aware Path and Motion Planning for a Tethered Aerial Visual Assistant in Unstructured or Confined Environments

This research aims at developing path and motion planning algorithms for a tethered Unmanned Aerial Vehicle (UAV) to visually assist a teleoperated primary robot in unstructured or confined environments. The emerging state of the practice for nuclear operations, bomb squad, disaster robots, and other domains with novel tasks or highly occluded environments is to use two robots, a primary and a secondary that acts as a visual assistant to overcome the perceptual limitations of the sensors by providing an external viewpoint. However, the benefits of using an assistant have been limited for at least three reasons: (1) users tend to choose suboptimal viewpoints, (2) only ground robot assistants are considered, ignoring the rapid evolution of small unmanned aerial systems for indoor flying, (3) introducing a whole crew for the second teleoperated robot is not cost effective, may introduce further teamwork demands, and therefore could lead to miscommunication. This dissertation proposes to use an autonomous tethered aerial visual assistant to replace the secondary robot and its operating crew. Along with a pre-established theory of viewpoint quality based on affordances, this dissertation aims at defining and representing robot motion risk in unstructured or confined environments. Based on those theories, a novel high level path planning algorithm is developed to enable risk-aware planning, which balances the tradeoff between viewpoint quality and motion risk in order to provide safe and trustworthy visual assistance flight. The planned flight trajectory is then realized on a tethered UAV platform. The perception and actuation are tailored to fit the tethered agent in the form of a low level motion suite, including a novel tether-based localization model with negligible computational overhead, motion primitives for the tethered airframe based on position and velocity control, and two different approaches to negotiate tether with complex obstacle-occupied environments. The proposed research provides a formal reasoning of motion risk in unstructured or confined spaces, contributes to the field of risk-aware planning with a versatile planner, and opens up a new regime of indoor UAV navigation: tethered indoor flight to ensure battery duration and failsafe in case of vehicle malfunction. It is expected to increase teleoperation productivity and reduce costly errors in scenarios such as safe decommissioning and nuclear operations in the Fukushima Daiichi facility

Modeling, Control, and Motion Analysis of a Class of Extensible Continuum Manipulators

In this dissertation, the development of a kinematic model, a configuration-space controller, a master-slave teleoperation controller, along with the analysis of the self-motion properties for redundant, extensible, continuous backbone (continuum) ``trunk and tentacle\u27 manipulators are detailed. Unlike conventional rigid-link robots, continuum manipulators are robots that can bend at any point along their backbone, resulting in new and unique modeling and control issues. Taken together, these chapters represent one of the first efforts towards devising model-based controllers of such robots, as well as characterizing their self-motion in its simplest form. Chapter 2 describes the development of a convenient set of generalized, spatial forward kinematics for extensible continuum manipulators based on the robot\u27s measurable variables. This development, takes advantage of the standard constant curvature assumption made for such manipulators and is simpler and more intuitive than the existing kinematic derivations which utilize a pseudo-rigid link manipulator. In Chapter 3, a new control strategy for continuum robots is presented. Control of this emerging new class of robots has proved difficult due to the inherent complexity of their dynamics. Using a recently established full Lagrangian dynamic model, a new nonlinear model-based control strategy (sliding-mode control) for continuum robots is introduced. Simulation results are illustrated using the dynamic model of a three-section, six Degree-of-Freedom, planar continuum robot and an experiment was conducted on the OctArm 9 Degree-of-Freedom continuum manipulator. In both the simulation and experiment, the results of the sliding-mode controller were found to be significantly better than a standard inverse-dynamics PD controller. In Chapter 4, the nature of continuum manipulator self-motion is studied. While use of the redundant continuum manipulator self-motion property (configuration changes which leave the end-effector location fixed) has been proposed, the nature of their null-spaces has not previously been explored. The manipulator related resolved-motion rate inverse kinematics which are based on the forward kinematics described in Chapter 2, are used. Based on these derivations, the self-motion of a 2-section, extensible redundant continuum manipulator in planar and spatial situations (generalizable to n-sections) is analyzed. The existence of a single self-motion manifold underlying the structures is proven, and simple self-motion cases spanning the null-space are introduced. The results of this analysis allow for a better understanding of general continuum robot self-motions and relate their underlying structure to real world examples and applications. The results are supported by experimental validation of the self-motion properties on the 9 Degree-of-Freedom OctArm continuum manipulator. In Chapter 5, teleoperation control of a kinematically redundant, continuum slave robot by a non-redundant, rigid-link master system is described. This problem is novel because the self-motion of the redundant robot can be utilized to achieve secondary control objectives while allowing the user to only control the tip of the slave system. To that end, feedback linearizing controllers are proposed for both the master and slave systems, whose effectiveness is demonstrated using numerical simulations and experimental results (using the 9 Degree-of-Freedom OctArm continuum manipulator as the slave system) for trajectory tracking as well as singularity avoidance subtask

Task priority control of underwater intervention systems: Theory and applications

This paper presents a unifying task priority control architecture for underwater vehicle manipulator systems. The proposed control framework can be applied to different operative scenarios such as waypoint navigation, assisted teleoperation, interaction, landing and grasping. This work extends the results of the TRIDENT and MARIS projects, which were limited to the execution of grasping actions, to other applications taken from the DexROV and ROBUST projects. In particular, simulation results show how the control framework can be used, for example, for pipeline inspection scenarios and deep sea mining exploration

Development of a twisted-string actuator for a cable-driven haptic interface

Il seguente lavoro di tesi ha avuto lo scopo di analizzare dal punto di vista della progettazione software le procedure ed i protocolli di scambio di informazioni tra i vari componenti di una nuova concezione di Interfaccia Aptica guidata da 4 tendini che muovono un braccialetto centrale collegato al braccio di un operatore così da fornirgli un feedback di forza. In particolare ci si è focalizzati sullo sviluppo di un firmware applicabile ai 4 motori che muovono la struttura centrale della interfaccia. Il firmware deve essere in grado di ricevere da una piattaforma Ros, usata da un operatore, pacchetti di dati contenenti i set point per i vari motori e il tipo di controllo , posizione o forza, che gli attuatori devono effettuare grazie ad uno schema PID. Inoltre l'invio di feedback all'operatore è stato previsto in modo da permettere una maggiore supervisione dell'intero funzionamento. La realizzazione di un Ros Bridge tra l'utente e il sistema da comandare è stato implementato con la formula della programmazione ad oggetti in cui varie classi sono dedicate a compiti differenti come l'impacchettamento di dati da mandare ai motori e la contemporanea ricezione dei feedback. Per completare tutta l'architettura si è anche sviluppato un sistema di trasformazione dei set point provenienti dall'operatore espressi nello spazio di lavoro Cartesiano in riferimenti per i singoli motori e ciò è stato possibile sfruttando la matrice Jacobiana. Una particolare attenzione è stata data all'aspetto di comunicazione dei dati e per fare ciò si è dovuta usare una architettura di codice a multithread e un protocollo UDP

A Cartesian Space Approach to Teleoperate a Slave Robot with a Kinematically Dissimilar Redundant Manipulator

Due to the inability of humans to interact with certain unstructured environments,telemanipulation of robots have gained immense importance. One of the primary tasks in telemanipulating robots remotely, is the effective manipulation of the slave robot using the master manipulator. Ideally a kinematic replica of the slave manipulator is used as the master to provide a joint-to-joint control to the slave. This research uses the 7-DOF Whole Arm Manipulator© (WAM) as the master manipulator and a 6-DOF Titan as the slave manipulator. Due to the kinematic dissimilarity between the two, a Cartesian space position mapping technique is adapted in which the slave is made to follow the same trajectory as the end effector of the master with respect to its reference frame. The main criterion in undertaking this mapping approach is to provide a convenient region of operation to the human operator. Various methods like pseudo inverse, Jacobian transpose and Damped least squares have been used to perform the inverse kinematics for the Titan. Joint limit avoidance and obstacle avoidance constraints were used to perform the inverse kinematics for the WAM and thereby remove the redundancy. Finally a joint volume limitation constraint (JVLC) was adopted which aims at providing the operator, a comfortable operational space in union with the master manipulator. Each inverse methodfor the Titan was experimentally tested and the best method identified from thesimulation results and the error analysis. Various experiments were also performed for the constrained inverse kinematics for the WAM and results were simulated. RoboWorks© was used for simulation purposes