Model-Augmented Haptic Telemanipulation: Concept, Retrospective Overview, and Current Use Cases

Certain telerobotic applications, including telerobotics in space, pose particularly demanding challenges to both technology and humans. Traditional bilateral telemanipulation approaches often cannot be used in such applications due to technical and physical limitations such as long and varying delays, packet loss, and limited bandwidth, as well as high reliability, precision, and task duration requirements. In order to close this gap, we research model-augmented haptic telemanipulation (MATM) that uses two kinds of models: a remote model that enables shared autonomous functionality of the teleoperated robot, and a local model that aims to generate assistive augmented haptic feedback for the human operator. Several technological methods that form the backbone of the MATM approach have already been successfully demonstrated in accomplished telerobotic space missions. On this basis, we have applied our approach in more recent research to applications in the fields of orbital robotics, telesurgery, caregiving, and telenavigation. In the course of this work, we have advanced specific aspects of the approach that were of particular importance for each respective application, especially shared autonomy, and haptic augmentation. This overview paper discusses the MATM approach in detail, presents the latest research results of the various technologies encompassed within this approach, provides a retrospective of DLR's telerobotic space missions, demonstrates the broad application potential of MATM based on the aforementioned use cases, and outlines lessons learned and open challenges

Model Mediated Teleoperation with a Hand-Arm Exoskeleton in Long Time Delays Using Reinforcement Learning

Telerobotic systems must adapt to new environmental conditions and deal with high uncertainty caused by long-time delays. As one of the best alternatives to human-level intelligence, Reinforcement Learning (RL) may offer a solution to cope with these issues. This paper proposes to integrate RL with the Model Mediated Teleoperation (MMT) concept. The teleoperator interacts with a simulated virtual environment, which provides instant feedback. Whereas feedback from the real environment is delayed, feedback from the model is instantaneous, leading to high transparency. The MMT is realized in combination with an intelligent system with two layers. The first layer utilizes Dynamic Movement Primitives (DMP) which accounts for certain changes in the avatar environment. And, the second layer addresses the problems caused by uncertainty in the model using RL methods. Augmented reality was also provided to fuse the avatar device and virtual environment models for the teleoperator. Implemented on DLR's Exodex Adam hand-arm haptic exoskeleton, the results show RL methods are able to find different solutions when changes are applied to the object position after the demonstration. The results also show DMPs to be effective at adapting to new conditions where there is no uncertainty involved

Model Mediated Teleoperation with a Hand-Arm Exoskeleton in Long Time Delays Using Reinforcement Learning

elerobotic systems must adapt to new environmental conditions and deal with high uncertainty caused by long-time delays. As one of the best alternatives to human-level intelligence, Reinforcement Learning (RL) may offer a solution to cope with these issues. This paper proposes to integrate RL with the Model Mediated Teleoperation (MMT) concept. The teleoperator interacts with a simulated virtual environment, which provides instant feedback. Whereas feedback from the real environment is delayed, feedback from the model is instantaneous, leading to high transparency. The MMT is realized in combination with an intelligent system with two layers. The first layer utilizes Dynamic Movement Primitives (DMP) which accounts for certain changes in the avatar environment. And, the second layer addresses the problems caused by uncertainty in the model using RL methods. Augmented reality was also provided to fuse the avatar device and virtual environment models for the teleoperator. Implemented on DLR's Exodex Adam hand-arm haptic exoskeleton, the results show RL methods are able to find different solutions when changes are applied to the object position after the demonstration. The results also show DMPs to be effective at adapting to new conditions where there is no uncertainty involved

The Shape of Damping: Optimizing Damping Coefficients to Improve Transparency on Bilateral Telemanipulation

This thesis presents a novel optimization-based passivity control algorithm for hapticenabled bilateral teleoperation systems involving multiple degrees of freedom. In particular, in the context of energy-bounding control, the contribution focuses on the implementation of a passivity layer for an existing time-domain scheme, ensuring optimal transparency of the interaction along subsets of the environment space which are preponderant for the given task, while preserving the energy bounds required for passivity. The involved optimization problem is convex and amenable to real-time implementation. The effectiveness of the proposed design is validated via an experiment performed on a virtual teleoperated environment. The interplay between transparency and stability is a critical aspect in haptic-enabled bilateral teleoperation control. While it is important to present the user with the true impedance of the environment, destabilizing factors such as time delays, stiff environments, and a relaxed grasp on the master device may compromise the stability and safety of the system. Passivity has been exploited as one of the the main tools for providing sufficient conditions for stable teleoperation in several controller design approaches, such as the scattering algorithm, timedomain passivity control, energy bounding algorithm, and passive set position modulation. In this work it is presented an innovative energy-based approach, which builds upon existing time-domain passivity controllers, improving and extending their effectiveness and functionality. The set of damping coefficients are prioritized in each degree of freedom, the resulting transparency presents a realistic force feedback in comparison to the other directions. Thus, the prioritization takes effect using a quadratic programming algorithm to find the optimal values for the damping. Finally, the energy tanks approach on passivity control is a solution used to ensure stability in a system for robotics bilateral manipulation. The bilateral telemanipulation must maintain the principle of passivity in all moments to preserve the system\u2019s stability. This work presents a brief introduction to haptic devices as a master component on the telemanipulation chain; the end effector in the slave side is a representation of an interactive object within an environment having a force sensor as feedback signal. The whole interface is designed into a cross-platform framework named ROS, where the user interacts with the system. Experimental results are presented

A Comprehensive Survey of the Tactile Internet: State of the art and Research Directions

The Internet has made several giant leaps over the years, from a fixed to a mobile Internet, then to the Internet of Things, and now to a Tactile Internet. The Tactile Internet goes far beyond data, audio and video delivery over fixed and mobile networks, and even beyond allowing communication and collaboration among things. It is expected to enable haptic communication and allow skill set delivery over networks. Some examples of potential applications are tele-surgery, vehicle fleets, augmented reality and industrial process automation. Several papers already cover many of the Tactile Internet-related concepts and technologies, such as haptic codecs, applications, and supporting technologies. However, none of them offers a comprehensive survey of the Tactile Internet, including its architectures and algorithms. Furthermore, none of them provides a systematic and critical review of the existing solutions. To address these lacunae, we provide a comprehensive survey of the architectures and algorithms proposed to date for the Tactile Internet. In addition, we critically review them using a well-defined set of requirements and discuss some of the lessons learned as well as the most promising research directions

A Scalable, High-Performance, Real-Time Control Architecture with Application to Semi-Autonomous Teleoperation

A scalable and real-time capable infrastructure is required to enable high-performance control and haptic rendering of systems with many degrees-of-freedom. The specific platform that motivates this thesis work is the open research platform da Vinci ReResearch Kit (dVRK). For the system architecture, we propose a specialized IEEE-1394 (FireWire) broadcast protocol that takes advantage of broadcast and peer-to-peer transfers to minimize the number of transactions, and thus the software overhead, on the control PC, thereby enabling fast real-time control. It has also been extended to Ethernet via a novel Ethernet-to-FireWire bridge protocol. The software architecture consists of a distributed hardware interface layer, a real-time component-based software framework, and integration with the Robot Operating System (ROS). The architecture is scalable to support multiple active manipulators, reconfigurable to enable researchers to partition a full system into multiple independent subsystems, and extensible at all levels of control. This architecture has been applied to two semi-autonomous teleoperation applications. The first application is a suturing task in Robotic Minimally Invasive Surgery (RMIS), that includes the development of virtual fixtures for the needle passing and knot tying sub-tasks, with a multi-user study to verify their effectiveness. The second application concerns time-delayed teleoperation of a robotic arm for satellite servicing. The research contribution includes the development of a line virtual fixture with augmented reality, a test for different time delay configurations and a multi-user study that evaluates the effectiveness of the system

Evaluation of the effects of user profile and interface characteristics on performance during robotic teleoperation.

Programa de P?s-Gradua??o em Instrumenta??o, Controle e Automa??o de Processos de Minera??o. Departamento de Engenharia de Controle e Automa??o, Escola de Minas, Universidade Federal de Ouro Preto.In the mining industry, operator safety has always been a crucial point, and therefore demands attention. In addition, given the high investments made in the enterprise, the search for a consistent and e cient production process is also relevant. To increase productivity and safety, teleoperation techniques can be used to allow the operator to be removed from the risk areas and operate the equipment from a safe distance. The work presented in this master's thesis was developed at the Instituto Tecnol?gico Vale (ITV) as a part of a broader project that aims the development of a conceptual framework that enables the analysis and validation of the application of teleoperation techniques in mining equipment. These equipment could be either used for exploration, engineering or extraction. More speci cally, this framework will permit the analysis of relevant information for teleoperation. This work represents the initial step taken, which comprises the execution of proof-of-concept tests in laboratory using a robotic system. Di erent teleoperation interfaces schemes consisting of video and haptic devices are analyzed under di erent aspects and the user point of view is taken into account. The goal is to analyze the e ects of the force feedback and of di erent visual feedback during teleoperation and to evaluate which prede ned users characteristics can contribute to better performance during teleoperation. Experimental tests were performed using a commercial mobile robotic platform Seekur Jr and teleoperation interfaces consisting of the haptic devices Novint Falcon and Geomagic Touch. This platform can be programmed in such a way that it operates emulating a mining equipment. For practical application-oriented tests, di erent operations from a mining process can be used as application of study (e.g. explosive charging or excavators operations). For the purposes of this work, interfaces for the control of the robotic platform locomotion were analyzed. In order to evaluate the in uence of interface characteristics and users pro le on performance during teleoperation two experiments were conducted. The experiments consisted of guiding the robotic platform in a scenario containing obstacles, using the di erent interfaces. In this work, delays in communication during teleoperation were not addressed. Results show that the use of force feedback e ectively contributes for better performances in term of task execution time, number of collisions with obstacles and task completion in teleoperation activities. Moreover, additional visual information, such as the environment map, can also increase perception of obstacles.Na ind?stria da minera??o, a seguran?a do operador sempre foi um ponto crucial, demandando, portanto, cuidadosa aten??o. Al?m disso, dados os altos investimentos realizados no empreendimento, a busca por um processo de produ??o consistente e e ciente ? tamb?m relevante. Com o objetivo de aumentar a produtividade e a seguran?a nas opera??es dentro da minera??o, t?cnicas de teleopera??o podem ser utilizadas para permitir que o operador seja removido das ?reas de risco e opere o equipamento ? dist?ncia. Em ess?ncia, a teleopera??o significa que ? poss?vel haver uma separa??o f?sica entre a unidade de comando (sistema local) e a unidade de execu??o (sistema remoto), ainda mantendo a rela??o entre os dois. Isto ?, apesar da separa??o f?sica, o sistema local ? ainda capaz de comandar o sistema remoto. Idealmente, para que tal comando seja feito de maneira mais e caz poss?vel, deseja-se prover ao operador uma experi?ncia com o ambiente remoto de forma natural e realista, de tal forma que ele tenha a sensa??o de estar interagindo diretamente com o ambiente remoto, por?m sem as perturba??es t?picas desses ambientes (ru?do, vibra??es, calor, etc.) e com a possibilidade de amplia??o dessa percep??o. Com a utiliza??o da teleopera??o, ? poss?vel, al?m de reduzir os riscos a que o operador est? exposto em suas atividades, melhorar seu desempenho na opera??o destes equipamentos durante a realiza??o das tarefas. O trabalho apresentado nesta disserta??o de mestrado foi desenvolvido no Instituto Tecnol?gico Vale (ITV) como parte de um projeto mais amplo que visa o desenvolvimento de um arcabou?o que permita a an?lise e valida??o da aplica??o de t?cnicas de teleopera??o a equipamentos de minera??o. Estes equipamentos podem ser usados tanto para tarefas de explora??o, quanto de engenharia ou extra??o. Mais especificamente, este arcabou?o permitir? a an?lise de qual informa??o ? relevante e auxilia o operador durante a opera??o do equipamento. No ?mbito deste projeto, o trabalho descrito nesta disserta??o de mestrado representa o primeiro passo dado, que compreende a execu??o de testes de conceito em laborat?rio usando um sistema rob?tico. Diferentes interfaces de teleopera??o compostas por dispositivos h?pticos e de v?deo s?o analisadas diante de diferentes aspectos e o ponto de vista do usu?rio ? levado em considera??o. O objetivo ? analisar os efeitos da realimenta??o de for?a e da realimenta??o visual durante a realiza??o de tarefas de maneira teleoperada e avaliar quais caracter?sticas predefinidas dos operadores podem contribuir para um melhor desempenho durante a teleopera??o de um equipamento. Para realizar os testes, a plataforma rob?tica m?vel comercial Seekur Jr e interfaces de teleopera??o compostas por dispositivos h?pticos Novint Falcon e Geomagic Touch s?o utilizados. Esta plataforma rob?tica pode ser programada de tal forma a operar emulando um equipamento de minera??o. Para a realiza??o de testes voltados a aplica??es pr?ticas, diferentes opera??es de um processo de minera??o podem ser utilizadas como aplica??o de estudo (carregamento de explosivos, opera??o de uma escavadeira, por exemplo). Para os fins de estudo deste trabalho, foram analisadas as interfaces para o controle da locomo??o da plataforma rob?tica. Para avaliar a influ?ncia das caracter?sticas da interface e do perfil dos usu?rios no desempenho durante a teleopera??o dois experimentos foram conduzidos. Os experimentos consistiram em comandar a plataforma rob?tica em um cen?rio contendo obst?culos utilizando as diferentes interfaces desenvolvidas. Neste trabalho, a presen?a de atrasos na comunica??o durante a teleopera??o n?o foi considerada. Os resultados mostram que o uso da realimenta??o de for?a efetivamente contribui para melhor desempenho durante a execu??o de tarefas de forma teleoperada, em termos do tempo de execu??o da tarefa, do n?mero de obst?culos colididos e do n?vel de conclus?o da tarefa. Al?m disso, informa??es visuais adicionais tais como um mapa do ambiente aumentam a percep??o de obst?culos

Enhancing tele-operation - Investigating the effect of sensory feedback on performance

The decline in the number of healthcare service providers in comparison to the growing numbers of service users prompts the development of technologies to improve the efficiency of healthcare services. One such technology which could offer support are assistive robots, remotely tele-operated to provide assistive care and support for older adults with assistive care needs and people living with disabilities. Tele-operation makes it possible to provide human-in-the-loop robotic assistance while also addressing safety concerns in the use of autonomous robots around humans. Unlike many other applications of robot tele-operation, safety is particularly significant as the tele-operated assistive robots will be used in close proximity to vulnerable human users. It is therefore important to provide as much information about the robot (and the robot workspace) as possible to the tele-operators to ensure safety, as well as efficiency. Since robot tele-operation is relatively unexplored in the context of assisted living, this thesis explores different feedback modalities that may be employed to communicate sensor information to tele-operators. The thesis presents research as it transitioned from identifying and evaluating additional feedback modalities that may be used to supplement video feedback, to exploring different strategies for communicating the different feedback modalities. Due to the fact that some of the sensors and feedback needed are not readily available, different design iterations were carried out to develop the necessary hardware and software for the studies carried out. The first human study was carried out to investigate the effect of feedback on tele-operator performance. Performance was measured in terms of task completion time, ease of use of the system, number of robot joint movements, and success or failure of the task. The effect of verbal feedback between the tele-operator and service users was also investigated. Feedback modalities have differing effects on performance metrics and as a result, the choice of optimal feedback may vary from task to task. Results show that participants preferred scenarios with verbal feedback relative to scenarios without verbal feedback, which also reflects in their performance. Gaze metrics from the study also showed that it may be possible to understand how tele-operators interact with the system based on their areas of interest as they carry out tasks. This findings suggest that such studies can be used to improve the design of tele-operation systems.The need for social interaction between the tele-operator and service user suggests that visual and auditory feedback modalities will be engaged as tasks are carried out. This further reduces the number of available sensory modalities through which information can be communicated to tele-operators. A wrist-worn Wi-Fi enabled haptic feedback device was therefore developed and a study was carried out to investigate haptic sensitivities across the wrist. Results suggest that different locations on the wrist have varying sensitivities to haptic stimulation with and without video distraction, duration of haptic stimulation, and varying amplitudes of stimulation. This suggests that dynamic control of haptic feedback can be used to improve haptic perception across the wrist, and it may also be possible to display more than one type of sensor data to tele-operators during a task. The final study carried out was designed to investigate if participants can differentiate between different types of sensor data conveyed through different locations on the wrist via haptic feedback. The effect of increased number of attempts on performance was also investigated. Total task completion time decreased with task repetition. Participants with prior gaming and robot experience had a more significant reduction in total task completion time when compared to participants without prior gaming and robot experience. Reduction in task completion time was noticed for all stages of the task but participants with additional feedback had higher task completion time than participants without supplementary feedback. Reduction in task completion time varied for different stages of the task. Even though gripper trajectory reduced with task repetition, participants with supplementary feedback had longer gripper trajectories than participants without supplementary feedback, while participants with prior gaming experience had shorter gripper trajectories than participants without prior gaming experience. Perceived workload was also found to reduce with task repetition but perceived workload was higher for participants with feedback reported higher perceived workload than participants without feedback. However participants without feedback reported higher frustration than participants without feedback.Results show that the effect of feedback may not be significant where participants can get necessary information from video feedback. However, participants were fully dependent on feedback when video feedback could not provide requisite information needed.The findings presented in this thesis have potential applications in healthcare, and other applications of robot tele-operation and feedback. Findings can be used to improve feedback designs for tele-operation systems to ensure safe and efficient tele-operation. The thesis also provides ways visual feedback can be used with other feedback modalities. The haptic feedback designed in this research may also be used to provide situational awareness for the visually impaired

Advances in Human-Robot Interaction

Rapid advances in the field of robotics have made it possible to use robots not just in industrial automation but also in entertainment, rehabilitation, and home service. Since robots will likely affect many aspects of human existence, fundamental questions of human-robot interaction must be formulated and, if at all possible, resolved. Some of these questions are addressed in this collection of papers by leading HRI researchers

Factories of the Future

Engineering; Industrial engineering; Production engineerin