Model-Augmented Haptic Telemanipulation: Concept, Retrospective Overview, and Current Use Cases

Certain telerobotic applications, including telerobotics in space, pose particularly demanding challenges to both technology and humans. Traditional bilateral telemanipulation approaches often cannot be used in such applications due to technical and physical limitations such as long and varying delays, packet loss, and limited bandwidth, as well as high reliability, precision, and task duration requirements. In order to close this gap, we research model-augmented haptic telemanipulation (MATM) that uses two kinds of models: a remote model that enables shared autonomous functionality of the teleoperated robot, and a local model that aims to generate assistive augmented haptic feedback for the human operator. Several technological methods that form the backbone of the MATM approach have already been successfully demonstrated in accomplished telerobotic space missions. On this basis, we have applied our approach in more recent research to applications in the fields of orbital robotics, telesurgery, caregiving, and telenavigation. In the course of this work, we have advanced specific aspects of the approach that were of particular importance for each respective application, especially shared autonomy, and haptic augmentation. This overview paper discusses the MATM approach in detail, presents the latest research results of the various technologies encompassed within this approach, provides a retrospective of DLR's telerobotic space missions, demonstrates the broad application potential of MATM based on the aforementioned use cases, and outlines lessons learned and open challenges

Space Science Opportunities Augmented by Exploration Telepresence

Since the end of the Apollo missions to the lunar surface in December 1972, humanity has exclusively conducted scientific studies on distant planetary surfaces using teleprogrammed robots. Operations and science return for all of these missions are constrained by two issues related to the great distances between terrestrial scientists and their exploration targets: high communication latencies and limited data bandwidth. Despite the proven successes of in-situ science being conducted using teleprogrammed robotic assets such as Spirit, Opportunity, and Curiosity rovers on the surface of Mars, future planetary field research may substantially overcome latency and bandwidth constraints by employing a variety of alternative strategies that could involve: 1) placing scientists/astronauts directly on planetary surfaces, as was done in the Apollo era; 2) developing fully autonomous robotic systems capable of conducting in-situ field science research; or 3) teleoperation of robotic assets by humans sufficiently proximal to the exploration targets to drastically reduce latencies and significantly increase bandwidth, thereby achieving effective human telepresence. This third strategy has been the focus of experts in telerobotics, telepresence, planetary science, and human spaceflight during two workshops held from October 3–7, 2016, and July 7–13, 2017, at the Keck Institute for Space Studies (KISS). Based on findings from these workshops, this document describes the conceptual and practical foundations of low-latency telepresence (LLT), opportunities for using derivative approaches for scientific exploration of planetary surfaces, and circumstances under which employing telepresence would be especially productive for planetary science. An important finding of these workshops is the conclusion that there has been limited study of the advantages of planetary science via LLT. A major recommendation from these workshops is that space agencies such as NASA should substantially increase science return with greater investments in this promising strategy for human conduct at distant exploration sites

Space Science Opportunities Augmented by Exploration Telepresence

Since the end of the Apollo missions to the lunar surface in December 1972, humanity has exclusively conducted scientific studies on distant planetary surfaces using teleprogrammed robots. Operations and science return for all of these missions are constrained by two issues related to the great distances between terrestrial scientists and their exploration targets: high communication latencies and limited data bandwidth. Despite the proven successes of in-situ science being conducted using teleprogrammed robotic assets such as Spirit, Opportunity, and Curiosity rovers on the surface of Mars, future planetary field research may substantially overcome latency and bandwidth constraints by employing a variety of alternative strategies that could involve: 1) placing scientists/astronauts directly on planetary surfaces, as was done in the Apollo era; 2) developing fully autonomous robotic systems capable of conducting in-situ field science research; or 3) teleoperation of robotic assets by humans sufficiently proximal to the exploration targets to drastically reduce latencies and significantly increase bandwidth, thereby achieving effective human telepresence. This third strategy has been the focus of experts in telerobotics, telepresence, planetary science, and human spaceflight during two workshops held from October 3–7, 2016, and July 7–13, 2017, at the Keck Institute for Space Studies (KISS). Based on findings from these workshops, this document describes the conceptual and practical foundations of low-latency telepresence (LLT), opportunities for using derivative approaches for scientific exploration of planetary surfaces, and circumstances under which employing telepresence would be especially productive for planetary science. An important finding of these workshops is the conclusion that there has been limited study of the advantages of planetary science via LLT. A major recommendation from these workshops is that space agencies such as NASA should substantially increase science return with greater investments in this promising strategy for human conduct at distant exploration sites

Appearance Modeling of Living Human Tissues

This is the peer reviewed version of the following article: Nunes, A.L.P., Maciel, A., Meyer, G.W., John, N.W., Baranoski, G.V.G., & Walter, M. (2019). Appearance Modeling of Living Human Tissues, Computer Graphics Forum, which has been published in final form at https://doi.org/10.1111/cgf.13604. This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Self-ArchivingThe visual fidelity of realistic renderings in Computer Graphics depends fundamentally upon how we model the appearance of objects resulting from the interaction between light and matter reaching the eye. In this paper, we survey the research addressing appearance modeling of living human tissue. Among the many classes of natural materials already researched in Computer Graphics, living human tissues such as blood and skin have recently seen an increase in attention from graphics research. There is already an incipient but substantial body of literature on this topic, but we also lack a structured review as presented here. We introduce a classification for the approaches using the four types of human tissues as classifiers. We show a growing trend of solutions that use first principles from Physics and Biology as fundamental knowledge upon which the models are built. The organic quality of visual results provided by these Biophysical approaches is mainly determined by the optical properties of biophysical components interacting with light. Beyond just picture making, these models can be used in predictive simulations, with the potential for impact in many other areas

Robot manipulator skill learning and generalising through teleoperation

Robot manipulators have been widely used for simple repetitive, and accurate tasks in industrial plants, such as pick and place, assembly and welding etc., but it is still hard to deploy in human-centred environments for dexterous manipulation tasks, such as medical examination and robot-assisted healthcare. These tasks are not only related to motion planning and control but also to the compliant interaction behaviour of robots, e.g. motion control, force regulation and impedance adaptation simultaneously under dynamic and unknown environments. Recently, with the development of collaborative robotics (cobots) and machine learning, robot skill learning and generalising have attained increasing attention from robotics, machine learning and neuroscience communities. Nevertheless, learning complex and compliant manipulation skills, such as manipulating deformable objects, scanning the human body and folding clothes, is still challenging for robots. On the other hand, teleoperation, also namely remote operation or telerobotics, has been an old research area since 1950, and there have been a number of applications such as space exploration, telemedicine, marine vehicles and emergency response etc. One of its advantages is to combine the precise control of robots with human intelligence to perform dexterous and safety-critical tasks from a distance. In addition, telepresence allows remote operators could feel the actual interaction between the robot and the environment, including the vision, sound and haptic feedback etc. Especially under the development of various augmented reality (AR), virtual reality (VR) and wearable devices, intuitive and immersive teleoperation have received increasing attention from robotics and computer science communities. Thus, various human-robot collaboration (HRC) interfaces based on the above technologies were developed to integrate robot control and telemanipulation by human operators for robot skills learning from human beings. In this context, robot skill learning could benefit teleoperation by automating repetitive and tedious tasks, and teleoperation demonstration and interaction by human teachers also allow the robot to learn progressively and interactively. Therefore, in this dissertation, we study human-robot skill transfer and generalising through intuitive teleoperation interfaces for contact-rich manipulation tasks, including medical examination, manipulating deformable objects, grasping soft objects and composite layup in manufacturing. The introduction, motivation and objectives of this thesis are introduced in Chapter 1. In Chapter 2, a literature review on manipulation skills acquisition through teleoperation is carried out, and the motivation and objectives of this thesis are discussed subsequently. Overall, the main contents of this thesis have three parts: Part 1 (Chapter 3) introduces the development and controller design of teleoperation systems with multimodal feedback, which is the foundation of this project for robot learning from human demonstration and interaction. In Part 2 (Chapters 4, 5, 6 and 7), we studied primitive skill library theory, behaviour tree-based modular method, and perception-enhanced method to improve the generalisation capability of learning from the human demonstrations. And several applications were employed to evaluate the effectiveness of these methods.In Part 3 (Chapter 8), we studied the deep multimodal neural networks to encode the manipulation skill, especially the multimodal perception information. This part conducted physical experiments on robot-assisted ultrasound scanning applications.Chapter 9 summarises the contributions and potential directions of this thesis. Keywords: Learning from demonstration; Teleoperation; Multimodal interface; Human-in-the-loop; Compliant control; Human-robot interaction; Robot-assisted sonography

Seventh Annual Workshop on Space Operations Applications and Research (SOAR 1993), volume 1

This document contains papers presented at the Space Operations, Applications and Research Symposium (SOAR) Symposium hosted by NASA/Johnson Space Center (JSC) on August 3-5, 1993, and held at JSC Gilruth Recreation Center. SOAR included NASA and USAF programmatic overview, plenary session, panel discussions, panel sessions, and exhibits. It invited technical papers in support of U.S. Army, U.S. Navy, Department of Energy, NASA, and USAF programs in the following areas: robotics and telepresence, automation and intelligent systems, human factors, life support, and space maintenance and servicing. SOAR was concerned with Government-sponsored research and development relevant to aerospace operations. More than 100 technical papers, 17 exhibits, a plenary session, several panel discussions, and several keynote speeches were included in SOAR '93

Remote tactile feedback on interactive surfaces

Direct touch input on interactive surfaces has become a predominating standard for the manipulation of digital information in our everyday lives. However, compared to our rich interchange with the physical world, the interaction with touch-based systems is limited in terms of flexibility of input and expressiveness of output. Particularly, the lack of tactile feedback greatly reduces the general usability of a touch-based system and hinders from a productive entanglement of the virtual information with the physical world. This thesis proposes remote tactile feedback as a novel method to provide programmed tactile stimuli supporting direct touch interactions. The overall principle is to spatially decouple the location of touch input (e.g. fingertip or hand) and the location of the tactile sensation on the user's body (e.g. forearm or back). Remote tactile feedback is an alternative concept which avoids particular challenges of existing approaches. Moreover, the principle provides inherent characteristics which can accommodate for the requirements of current and future touch interfaces. To define the design space, the thesis provides a structured overview of current forms of touch surfaces and identifies trends towards non-planar and non-rigid forms with more versatile input mechanisms. Furthermore, a classification highlights limitations of the current methods to generate tactile feedback on touch-based systems. The proposed notion of tactile sensory relocation is a form of sensory substitution. Underlying neurological and psychological principles corroborate the approach. Thus, characteristics of the human sense of touch and principles from sensory substitution help to create a technical and conceptual framework for remote tactile feedback. Three consecutive user studies measure and compare the effects of both direct and remote tactile feedback on the performance and the subjective ratings of the user. Furthermore, the experiments investigate different body locations for the application of tactile stimuli. The results show high subjective preferences for tactile feedback, regardless of its type of application. Additionally, the data reveals no significant differences between the effects of direct and remote stimuli. The results back the feasibility of the approach and provide parameters for the design of stimuli and the effective use of the concept. The main part of the thesis describes the systematical exploration and analysis of the inherent characteristics of remote tactile feedback. Four specific features of the principle are identified: (1) the simplification of the integration of cutaneous stimuli, (2) the transmission of proactive, reactive and detached feedback, (3) the increased expressiveness of tactile sensations and (4) the provision of tactile feedback during multi-touch. In each class, several prototypical remote tactile interfaces are used in evaluations to analyze the concept. For example, the PhantomStation utilizes psychophysical phenomena to reduce the number of single tactile actuators. An evaluation with the prototype compares standard actuator technologies with each other in order to enable simple and scalable implementations. The ThermalTouch prototype creates remote thermal stimuli to reproduce material characteristics on standard touchscreens. The results show a stable rate of virtual object discrimination based on remotely applied temperature profiles. The AutmotiveRTF system is implemented in a vehicle and supports the driver's input on the in-vehicle-infotainment system. A field study with the system focuses on evaluating the effects of proactive and reactive feedback on the user's performance. The main contributions of the dissertation are: First, the thesis introduces the principle of remote tactile feedback and defines a design space for this approach as an alternative method to provide non-visual cues on interactive surfaces. Second, the thesis describes technical examples to rapidly prototype remote tactile feedback systems. Third, these prototypes are deployed in several evaluations which highlight the beneficial subjective and objective effects of the approach. Finally, the thesis presents features and inherent characteristics of remote tactile feedback as a means to support the interaction on today's touchscreens and future interactive surfaces.Die Interaktion mit berührungsempfindlichen Oberflächen ist heute ein Standard für die Manipulation von digitaler Information. Jedoch weist die Bedienung dieser interaktiven Bildschirme starke Einschränkungen hinsichtlich der Flexibilität bei der Eingabe und der Ausdruckskraft der Ausgabe auf, wenn man sie mit den vielfältigen Möglichkeiten des Umgangs mit Objekten in unserer Alltagswelt vergleicht. Besonders die nicht vorhandenen Tastsinnesrückmeldungen vermindern stark die Benutzbarkeit solcher Systeme und verhindern eine effektive Verknüpfung von virtueller Information und physischer Welt. Die vorliegende Dissertation beschreibt den Ansatz der 'distalen taktilen Rückmeldungen' als neuartige Möglichkeit zur Vermittlung programmierter Tastsinnesreize an Benutzer interaktiver Oberflächen. Das Grundprinzip dabei ist die räumliche Trennung zwischen der Eingabe durch Berührung (z.B. mit der Fingerspitze) und dem daraus resultierenden taktilen Reiz am Körper der Benutzer (z.B. am Rücken). Dabei vermeidet das Konzept der distalen taktilen Rückmeldungen einzelne technische und konzeptionelle Nachteile existierender Ansätze. Zusätzlich bringt es Interaktionsmöglichkeiten mit sich, die den Eigenheiten der Interaktion mit aktuellen und auch zukünftigen berührungsempfindlichen Oberflächen Rechnung tragen. Zu Beginn zeigt ein Überblick zu relevanten Arbeiten den aktuellen Forschungstrend hin zu nicht-flachen und verformbaren berührungsempfindlichen Oberflächen sowie zu vielfältigeren Eingabemethoden. Eine Klassifizierung ordnet existierende technische Verfahren zur Erzeugung von künstlichen Tastsinnesreizen und stellt jeweils konzeptuelle und technische Herausforderungen dar. Der in dieser Arbeit vorgeschlagene Ansatz der Verlagerung von Tastsinnesreizen ist eine Form der sensorischen Substitution, zugrunde liegende neurologische und psychologische Prinzipien untermauern das Vorgehen. Die Wirkprinzipien des menschlichen Tastsinnes und die Systeme zur sensorischen Substitution liefern daher konzeptionelle und technische Richtlinien zur Umsetzung der distalen taktilen Rückmeldungen. Drei aufeinander aufbauende Benutzerstudien vergleichen die Auswirkungen von direkten und distalen taktilen Rückmeldungen auf die Leistung und das Verhalten von Benutzern sowie deren subjektive Bewertung der Interaktion. Außerdem werden in den Experimenten die Effekte von Tastsinnesreizen an verschiedenen Körperstellen untersucht. Die Ergebnisse zeigen starke Präferenzen für Tastsinnesrückmeldungen, unabhängig von deren Applikationsort. Die Daten ergeben weiterhin keine signifikanten Unterschiede bei den quantitativen Effekten von direktem und distalen Rückmeldungen. Diese Ergebnisse befürworten die Realisierbarkeit des Ansatzes und zeigen Richtlinien für weitere praktische Umsetzungen auf. Der Hauptteil der Dissertation beschreibt die systematische Untersuchung und Analyse der inhärenten Möglichkeiten, die sich aus der Vermittlung distaler taktiler Rückmeldungen ergeben. Vier verschiedene Charakteristika werden identifiziert: (1) die vereinfachte Integration von Tastsinnesreizen, (2) die Vermittlung von proaktiven, reaktiven und entkoppelten Rückmeldungen, (3) die erhöhte Bandbreite der taktilen Signale und (4) die Darstellung von individuellen Tastsinnesreizen für verschiedene Kontaktpunkte mit der berührungsempfindlichen Oberfläche. Jedes dieser Prinzipien wird durch prototypische Systeme umgesetzt und in Benutzerstudien analysiert. Beispielsweise nutzt das System PhantomStation psychophysikalische Illusionen, um die Anzahl der einzelnen Reizgeber zu reduzieren. In einer Evaluierung des Prototypen werden mehrere Aktuatortechnologien verglichen, um einfache und skalierbare Ansätze zu identifizieren. Der ThermalTouch-Prototyp wird dazu genutzt, distale thermale Reize zu vermitteln, um so Materialeigenschaften auf Berührungsbildschirmen darstellen zu können. Eine Benutzerstudie zeigt, dass sich auf Basis dieser Temperaturverläufe virtuelle Objekte unterscheiden lassen. Das AutomotiveRTF-System wird schließlich in ein Kraftfahrzeug integriert, um den Fahrer bei der Eingabe auf dem Informations- und Unterhaltungssystem zu unterstützen. Eine Feldstudie untersucht die Auswirkungen der proaktiven und reaktiven Rückmeldungen auf die Benutzerleistung. Die vorliegende Dissertation leistet mehrere Beiträge zur Mensch-Maschine-Interaktion: Das Prinzip der distalen taktilen Rückmeldungen wird eingeführt als Alternative zur Erzeugung nicht-visueller Rückmeldungen auf interaktiven Oberflächen. Es werden technische Verfahrensweisen zur prototypischen Implementierung solcher Systeme vorgeschlagen. Diese technischen Prototypen werden in einer Vielzahl verschiedener Benutzerstudien eingesetzt, welche die quantitativen und qualitativen Vorteile des Ansatzes aufzeigen. Schließlich wird gezeigt, wie sich das Prinzip zur Unterstützung heutiger und zukünftiger Interaktionsformen mit berührungsempfindlichen Bildschirmen nutzen lässt