Multi-robot cooperative platform : a task-oriented teleoperation paradigm

This thesis proposes the study and development of a teleoperation system based on multi-robot cooperation under the task oriented teleoperation paradigm: Multi-Robot Cooperative Paradigm, MRCP. In standard teleoperation, the operator uses the master devices to control the remote slave robot arms. These arms reproduce the desired movements and perform the task. With the developed work, the operator can virtually manipulate an object. MRCP automatically generates the arms orders to perform the task. The operator does not have to solve situations arising from possible restrictions that the slave arms may have. The research carried out is therefore aimed at improving the accuracy teleoperation tasks in complex environments, particularly in the field of robot assisted minimally invasive surgery. This field requires patient safety and the workspace entails many restrictions to teleoperation. MRCP can be defined as a platform composed of several robots that cooperate automatically to perform a teleoperated task, creating a robotic system with increased capacity (workspace volume, accessibility, dexterity ...). The cooperation is based on transferring the task between robots when necessary to enable a smooth task execution. The MRCP control evaluates the suitability of each robot to continue with the ongoing task and the optimal time to execute a task transfer between the current selected robot and the best candidate to continue with the task. From the operator¿s point of view, MRCP provides an interface that enables the teleoperation though the task-oriented paradigm: operator orders are translated into task actions instead of robot orders. This thesis is structured as follows: The first part is dedicated to review the current solutions in the teleoperation of complex tasks and compare them with those proposed in this research. The second part of the thesis presents and reviews in depth the different evaluation criteria to determine the suitability of each robot to continue with the execution of a task, considering the configuration of the robots and emphasizing the criterion of dexterity and manipulability. The study reviews the different required control algorithms to enable the task oriented telemanipulation. This proposed teleoperation paradigm is transparent to the operator. Then, the Thesis presents and analyses several experimental results using MRCP in the field of minimally invasive surgery. These experiments study the effectiveness of MRCP in various tasks requiring the cooperation of two hands. A type task is used: a suture using minimally invasive surgery technique. The analysis is done in terms of execution time, economy of movement, quality and patient safety (potential damage produced by undesired interaction between the tools and the vital tissues of the patient). The final part of the thesis proposes the implementation of different virtual aids and restrictions (guided teleoperation based on haptic visual and audio feedback, protection of restricted workspace regions, etc.) using the task oriented teleoperation paradigm. A framework is defined for implementing and applying a basic set of virtual aids and constraints within the framework of a virtual simulator for laparoscopic abdominal surgery. The set of experiments have allowed to validate the developed work. The study revealed the influence of virtual aids in the learning process of laparoscopic techniques. It has also demonstrated the improvement of learning curves, which paves the way for its implementation as a methodology for training new surgeons.Aquesta tesi doctoral proposa l'estudi i desenvolupament d'un sistema de teleoperació basat en la cooperació multi-robot sota el paradigma de la teleoperació orientada a tasca: Multi-Robot Cooperative Paradigm, MRCP. En la teleoperació clàssica, l'operador utilitza els telecomandaments perquè els braços robots reprodueixin els seus moviments i es realitzi la tasca desitjada. Amb el treball realitzat, l'operador pot manipular virtualment un objecte i és mitjançant el MRCP que s'adjudica a cada braç les ordres necessàries per realitzar la tasca, sense que l'operador hagi de resoldre les situacions derivades de possibles restriccions que puguin tenir els braços executors. La recerca desenvolupada està doncs orientada a millorar la teleoperació en tasques de precisió en entorns complexos i, en particular, en el camp de la cirurgia mínimament invasiva assistida per robots. Aquest camp imposa condicions de seguretat del pacient i l'espai de treball comporta moltes restriccions a la teleoperació. MRCP es pot definir com a una plataforma formada per diversos robots que cooperen de forma automàtica per dur a terme una tasca teleoperada, generant un sistema robòtic amb capacitats augmentades (volums de treball, accessibilitat, destresa,...). La cooperació es basa en transferir la tasca entre robots a partir de determinar quin és aquell que és més adequat per continuar amb la seva execució i el moment òptim per realitzar la transferència de la tasca entre el robot actiu i el millor candidat a continuar-la. Des del punt de vista de l'operari, MRCP ofereix una interfície de teleoperació que permet la realització de la teleoperació mitjançant el paradigma d'ordres orientades a la tasca: les ordres es tradueixen en accions sobre la tasca en comptes d'estar dirigides als robots. Aquesta tesi està estructurada de la següent manera: Primerament es fa una revisió de l'estat actual de les diverses solucions desenvolupades actualment en el camp de la teleoperació de tasques complexes, comparant-les amb les proposades en aquest treball de recerca. En el segon bloc de la tesi es presenten i s'analitzen a fons els diversos criteris per determinar la capacitat de cada robot per continuar l'execució d'una tasca, segons la configuració del conjunt de robots i fent especial èmfasi en el criteri de destresa i manipulabilitat. Seguint aquest estudi, es presenten els diferents processos de control emprats per tal d'assolir la telemanipulació orientada a tasca de forma transparent a l'operari. Seguidament es presenten diversos resultats experimentals aplicant MRCP al camp de la cirurgia mínimament invasiva. En aquests experiments s'estudia l'eficàcia de MRCP en diverses tasques que requereixen de la cooperació de dues mans. S'ha escollit una tasca tipus: sutura amb tècnica de cirurgia mínimament invasiva. L'anàlisi es fa en termes de temps d'execució, economia de moviment, qualitat i seguretat del pacient (potencials danys causats per la interacció no desitjada entre les eines i els teixits vitals del pacient). Finalment s'ha estudiat l'ús de diferents ajudes i restriccions virtuals (guiat de la teleoperació via retorn hàptic, visual o auditiu, protecció de regions de l'espai de treball, etc) dins el paradigma de teleoperació orientada a tasca. S'ha definint un marc d'aplicació base i implementant un conjunt de restriccions virtuals dins el marc d'un simulador de cirurgia laparoscòpia abdominal. El conjunt d'experiments realitzats han permès validar el treball realitzat. Aquest estudi ha permès determinar la influencia de les ajudes virtuals en el procés d'aprenentatge de les tècniques laparoscòpiques. S'ha evidenciat una millora en les corbes d'aprenentatge i obre el camí a la seva implantació com a metodologia d'entrenament de nous cirurgians.Postprint (published version

Mobile robot transportation in laboratory automation

In this dissertation a new mobile robot transportation system is developed for the modern laboratory automation to connect the distributed automated systems and workbenches. In the system, a series of scientific and technical robot indoor issues are presented and solved, including the multiple robot control strategy, the indoor transportation path planning, the hybrid robot indoor localization, the recharging optimization, the robot-automated door interface, the robot blind arm grasping & placing, etc. The experiments show the proposed system and methods are effective and efficient

Virtual Reality Games for Motor Rehabilitation

This paper presents a fuzzy logic based method to track user satisfaction without the need for devices to monitor users physiological conditions. User satisfaction is the key to any product’s acceptance; computer applications and video games provide a unique opportunity to provide a tailored environment for each user to better suit their needs. We have implemented a non-adaptive fuzzy logic model of emotion, based on the emotional component of the Fuzzy Logic Adaptive Model of Emotion (FLAME) proposed by El-Nasr, to estimate player emotion in UnrealTournament 2004. In this paper we describe the implementation of this system and present the results of one of several play tests. Our research contradicts the current literature that suggests physiological measurements are needed. We show that it is possible to use a software only method to estimate user emotion

Kontextsensitive Körperregulierung für redundante Roboter

In the past few decades the classical 6 degrees of freedom manipulators' dominance has been challenged by the rise of 7 degrees of freedom redundant robots. Similarly, with increased availability of humanoid robots in academic research, roboticists suddenly have access to highly dexterous platforms with multiple kinematic chains capable of undertaking multiple tasks simultaneously. The execution of lower-priority tasks, however, are often done in task/scenario specific fashion. Consequently, these systems are not scalable and slight changes in the application often implies re-engineering the entire control system and deployment which impedes the development process over time. This thesis introduces an alternative systematic method of addressing the secondary tasks and redundancy resolution called, context aware body regulation. Contexts consist of one or multiple tasks, however, unlike the conventional definitions, the tasks within a context are not rigidly defined and maintain some level of abstraction. For instance, following a particular trajectory constitutes a concrete task while performing a Cartesian motion with the end-effector represents an abstraction of the same task and is more appropriate for context formulation. Furthermore, contexts are often made up of multiple abstract tasks that collectively describe a reoccurring situation. Body regulation is an umbrella term for a collection of schemes for addressing the robots' redundancy when a particular context occurs. Context aware body regulation offers several advantages over traditional methods. Most notably among them are reusability, scalability and composability of contexts and body regulation schemes. These three fundamental concerns are realized theoretically by in-depth study and through mathematical analysis of contexts and regulation strategies; and are practically implemented by a component based software architecture that complements the theoretical aspects. The findings of the thesis are applicable to any redundant manipulator and humanoids, and allow them to be used in real world applications. Proposed methodology presents an alternative approach for the control of robots and offers a new perspective for future deployment of robotic solutions.Im Verlauf der letzten Jahrzehnte wich der Einfluss klassischer Roboterarme mit 6 Freiheitsgraden zunehmend denen neuer und vielfältigerer Manipulatoren mit 7 Gelenken. Ebenso stehen der Forschung mit den neuartigen Humanoiden inzwischen auch hoch-redundante Roboterplattformen mit mehreren kinematischen Ketten zur Verfügung. Diese überaus flexiblen und komplexen Roboter-Kinematiken ermöglichen generell das gleichzeitige Verfolgen mehrerer priorisierter Bewegungsaufgaben. Die Steuerung der weniger wichtigen Aufgaben erfolgt jedoch oft in anwendungsspezifischer Art und Weise, welche die Skalierung der Regelung zu generellen Kontexten verhindert. Selbst kleine Änderungen in der Anwendung bewirken oft schon, dass große Teile der Robotersteuerung überarbeitet werden müssen, was wiederum den gesamten Entwicklungsprozess behindert. Diese Dissertation stellt eine alternative, systematische Methode vor um die Redundanz neuer komplexer Robotersysteme zu bewältigen und vielfältige, priorisierte Bewegungsaufgaben parallel zu steuern: Die so genannte kontextsensitive Körperregulierung. Darin bestehen Kontexte aus einer oder mehreren Bewegungsaufgaben. Anders als in konventionellen Anwendungen sind die Aufgaben nicht fest definiert und beinhalten eine gewisse Abstraktion. Beispielsweise stellt das Folgen einer bestimmten Trajektorie eine sehr konkrete Bewegungsaufgabe dar, während die Ausführung einer Kartesischen Bewegung mit dem Endeffektor eine Abstraktion darstellt, die für die Kontextformulierung besser geeignet ist. Kontexte setzen sich oft aus mehreren solcher abstrakten Aufgaben zusammen und beschreiben kollektiv eine sich wiederholende Situation. Durch die Verwendung der kontextsensitiven Körperregulierung ergeben sich vielfältige Vorteile gegenüber traditionellen Methoden: Wiederverwendbarkeit, Skalierbarkeit, sowie Komponierbarkeit von Konzepten. Diese drei fundamentalen Eigenschaften werden in der vorliegenden Arbeit theoretisch mittels gründlicher mathematischer Analyse aufgezeigt und praktisch mittels einer auf Komponenten basierenden Softwarearchitektur realisiert. Die Ergebnisse dieser Dissertation lassen sich auf beliebige redundante Manipulatoren oder humanoide Roboter anwenden und befähigen diese damit zur realen Anwendung außerhalb des Labors. Die hier vorgestellte Methode zur Regelung von Robotern stellt damit eine neue Perspektive für die zukünftige Entwicklung von robotischen Lösungen dar

2020 NASA Technology Taxonomy

This document is an update (new photos used) of the PDF version of the 2020 NASA Technology Taxonomy that will be available to download on the OCT Public Website. The updated 2020 NASA Technology Taxonomy, or "technology dictionary", uses a technology discipline based approach that realigns like-technologies independent of their application within the NASA mission portfolio. This tool is meant to serve as a common technology discipline-based communication tool across the agency and with its partners in other government agencies, academia, industry, and across the world

On Safe Usage of Shared Data in Safety-Critical Control Systems

Prognostiziert durch Konzepte der Industrie 4.0 und den Cyber-Physischen-Systemen, können autonome Systeme zukünftig dynamisch auf Datenquellen in ihrer Umgebung zugreifen. Während die gemeinsame Nutzung solcher Datenquellen ein enormes Performanzpotenzial bietet, stellt die benötigte Systemarchitektur vorherrschende Sicherheitsprozesse vor neue Herausforderungen. Die vorliegende Arbeit motiviert zunächst, dass diese nur zur Laufzeit des Systems adressiert werden könne, bevor sie daraus zwei zentrale Ziele ableitet und verfolgt. Zum einen wird ein Beschreibungsmodel für die Darstellung von Fehlercharakteristika gemeinsam genutzter Daten vorgestellt. Dieses generische Fehlermodell erlaubt es zum anderen eine Sicherheitsanalyse zu definieren, die eine spezifische, dynamische Systemkomposition zur Laufzeit mit Hinblick auf die zu erwartenden Unsicherheiten bewerten kann. Die als Region of Safety betitelte Analysestrategie erlaubt, in Kombination mit dem generischen Fehlermodell, die Sicherheit der auf gemeinsam genutzten Daten basierenden Kollisionsvermeidungsstrategie zweier Roboter noch zur Designzeit zu garantieren, obwohl die spezifischen Fehlercharakteristika der Daten erst zur Laufzeit bekannt werden.:List of Acronyms List of Theorems List of Definitions List of Figures List of Tables 1. Introduction – Safety in Future Smart Industries 1.1. The Example of Smart Warehouses 1.2. Functional Safety Standards 1.2.1. Overview of Functional Safety Standards 1.2.2. IEC 61508 1.3. Scope of this Thesis 1.3.1. Objectives 1.3.2. Contributions 1.3.3. Outline 1.4. Related Publications by the Author 1.5. Mathematical Notation 2. State of the Art 2.1. State of the Art in Run-Time Safety Assessment 2.1.1. Approaches at the Functional Level 2.1.2. Approaches at the Technical Level 2.1.3. Conclusions 2.2. State of the Art in Failure Modeling 2.2.1. The Definition of (Sensor) Failure Model 2.2.2. Interval-Based Failure Modeling 2.2.3. Distribution-Based Failure Modeling 2.2.4. Failure-Type-Based Failure Modeling 2.2.5. Conclusions 2.3. Conclusions from the State of the Art 3. Generic Failure Model 3.1. Defining the Generic Failure Model 3.1.1. Time- and Value-Correlated Random Distribution 3.1.2. A Failure Type’s Failure Amplitudes 3.1.3. A Failure Type’s State Function 3.1.4. Polynomial Representation of a Failure Type 3.1.5. Discussion on the Fulfillment of the Predefined Criteria 3.2. Converting a Generic Failure Model to an Interval 3.2.1. Converting a Time- and Value-Correlated Random Distribution 3.2.2. A Failure Type’s Interval 3.3. Processing Chain for Generating Generic Failure Models 3.3.1. Identifying Failure Types 3.3.2. Parameterizing Failure Types 3.3.3. Confidence Calculation 3.4. Exemplary Application to Artificial Failure Characteristics 3.4.1. Generating the Artificial Data Set – Manually Designing GFMs 3.4.2. Identifying Failure Types 3.4.3. Parameterizing Failure Types 3.4.4. Confidence Calculation 3.4.5. Comparison to State-of-the-Art Models 3.5. Summary 4. Region of Safety 4.1. Explicitly Modeling Uncertainties for Dynamically Composed Systems 4.2. Regions of Safety for Dynamically Composed Systems 4.2.1. Estimating Regions of Attraction in Presence of Uncertainty 4.2.2. Introducing the Concept of Region of Safety 4.2.3. Discussion on the Fulfillment of the Predefined Criteria 4.3. Evaluating the Concept of Region of Safety 4.3.1. Defining the Scenario and Considered Uncertainties 4.3.2. Designing a Control Lyapunov Function 4.3.3. Determining an Appropriate Value for λc 4.3.4. The Effect of Varying Sensor Failures on Regions of Safety 4.4. Summary 5. Evaluation and Integration 5.1. Multi-Robot Collision Avoidance 5.1.1. Assumptions 5.1.2. Design of the Circle and Navigation Scenarios 5.1.3. Kinematics 5.1.4. Control Policy 5.1.5. Intention Modeling by Model Uncertainty 5.1.6. Fusing Regions of Safety of Multiple Stability Points 5.2. Failure Modeling for Shared Data – A Marker Detection Failure Model 5.2.1. Data Acquisition 5.2.2. Failure Model Generation 5.2.3. Evaluating the Quality of the Failure Model 5.3. Safe Handling of Shared Data in a Collision Avoidance Strategy 5.3.1. Configuration for Region of Safety Estimation 5.3.2. Estimating Regions of Safety 5.3.3. Evaluation Using the Circle Scenario 5.3.4. Evaluation Using the Navigation Scenario 5.4. Summary 6. Conclusions and Future Work 6.1. Summary 6.2. Limitations and Future Work 6.2.1. Limitations and Future Work on the Generic Failure Model 6.2.2. Limitations and Future Work on Region of Safety 6.2.3. Future Work on Safety in Dynamically Composed Systems Appendices A. Defining Factors of Risk According to IEC 61508 B. Evaluation Results for the Identification Stage C. Overview of Failure Amplitudes of Marker Detection Results BibliographyThe concepts of Cyber-Physical-Systems and Industry 4.0 prognosticate autonomous systems to integrate sources of shared data dynamically at their run-time. While this promises substantial increases in their performance, the openness of the required system architecture poses new challenges to processes guaranteeing their safety. This thesis firstly motivates that these can be addressed only at their run-time, before it derives and pursues two corresponding goals. Firstly, a model for describing failure characteristics of shared data is presented. Secondly, this Generic Failure Model is built upon to define a run-time safety assessment methodology that enables analyzing dynamic system compositions integrating shared data with respect to the expected uncertainties at run-time. This analysis strategy, entitled Region of Safety, allows in combination with the generic failure model to guarantee the safety of robots sharing position data for collision avoidance already at design-time, although specific failure characteristics become available only at run-time.:List of Acronyms List of Theorems List of Definitions List of Figures List of Tables 1. Introduction – Safety in Future Smart Industries 1.1. The Example of Smart Warehouses 1.2. Functional Safety Standards 1.2.1. Overview of Functional Safety Standards 1.2.2. IEC 61508 1.3. Scope of this Thesis 1.3.1. Objectives 1.3.2. Contributions 1.3.3. Outline 1.4. Related Publications by the Author 1.5. Mathematical Notation 2. State of the Art 2.1. State of the Art in Run-Time Safety Assessment 2.1.1. Approaches at the Functional Level 2.1.2. Approaches at the Technical Level 2.1.3. Conclusions 2.2. State of the Art in Failure Modeling 2.2.1. The Definition of (Sensor) Failure Model 2.2.2. Interval-Based Failure Modeling 2.2.3. Distribution-Based Failure Modeling 2.2.4. Failure-Type-Based Failure Modeling 2.2.5. Conclusions 2.3. Conclusions from the State of the Art 3. Generic Failure Model 3.1. Defining the Generic Failure Model 3.1.1. Time- and Value-Correlated Random Distribution 3.1.2. A Failure Type’s Failure Amplitudes 3.1.3. A Failure Type’s State Function 3.1.4. Polynomial Representation of a Failure Type 3.1.5. Discussion on the Fulfillment of the Predefined Criteria 3.2. Converting a Generic Failure Model to an Interval 3.2.1. Converting a Time- and Value-Correlated Random Distribution 3.2.2. A Failure Type’s Interval 3.3. Processing Chain for Generating Generic Failure Models 3.3.1. Identifying Failure Types 3.3.2. Parameterizing Failure Types 3.3.3. Confidence Calculation 3.4. Exemplary Application to Artificial Failure Characteristics 3.4.1. Generating the Artificial Data Set – Manually Designing GFMs 3.4.2. Identifying Failure Types 3.4.3. Parameterizing Failure Types 3.4.4. Confidence Calculation 3.4.5. Comparison to State-of-the-Art Models 3.5. Summary 4. Region of Safety 4.1. Explicitly Modeling Uncertainties for Dynamically Composed Systems 4.2. Regions of Safety for Dynamically Composed Systems 4.2.1. Estimating Regions of Attraction in Presence of Uncertainty 4.2.2. Introducing the Concept of Region of Safety 4.2.3. Discussion on the Fulfillment of the Predefined Criteria 4.3. Evaluating the Concept of Region of Safety 4.3.1. Defining the Scenario and Considered Uncertainties 4.3.2. Designing a Control Lyapunov Function 4.3.3. Determining an Appropriate Value for λc 4.3.4. The Effect of Varying Sensor Failures on Regions of Safety 4.4. Summary 5. Evaluation and Integration 5.1. Multi-Robot Collision Avoidance 5.1.1. Assumptions 5.1.2. Design of the Circle and Navigation Scenarios 5.1.3. Kinematics 5.1.4. Control Policy 5.1.5. Intention Modeling by Model Uncertainty 5.1.6. Fusing Regions of Safety of Multiple Stability Points 5.2. Failure Modeling for Shared Data – A Marker Detection Failure Model 5.2.1. Data Acquisition 5.2.2. Failure Model Generation 5.2.3. Evaluating the Quality of the Failure Model 5.3. Safe Handling of Shared Data in a Collision Avoidance Strategy 5.3.1. Configuration for Region of Safety Estimation 5.3.2. Estimating Regions of Safety 5.3.3. Evaluation Using the Circle Scenario 5.3.4. Evaluation Using the Navigation Scenario 5.4. Summary 6. Conclusions and Future Work 6.1. Summary 6.2. Limitations and Future Work 6.2.1. Limitations and Future Work on the Generic Failure Model 6.2.2. Limitations and Future Work on Region of Safety 6.2.3. Future Work on Safety in Dynamically Composed Systems Appendices A. Defining Factors of Risk According to IEC 61508 B. Evaluation Results for the Identification Stage C. Overview of Failure Amplitudes of Marker Detection Results Bibliograph

Annals of Scientific Society for Assembly, Handling and Industrial Robotics 2021

This Open Access proceedings presents a good overview of the current research landscape of assembly, handling and industrial robotics. The objective of MHI Colloquium is the successful networking at both academic and management level. Thereby, the colloquium focuses an academic exchange at a high level in order to distribute the obtained research results, to determine synergy effects and trends, to connect the actors in person and in conclusion, to strengthen the research field as well as the MHI community. In addition, there is the possibility to become acquatined with the organizing institute. Primary audience is formed by members of the scientific society for assembly, handling and industrial robotics (WGMHI)