Expert-in-the-Loop Multilateral Telerobotics for Haptics-Enabled Motor Function and Skills Development

Among medical robotics applications are Robotics-Assisted Mirror Rehabilitation Therapy (RAMRT) and Minimally-Invasive Surgical Training (RAMIST) that extensively rely on motor function development. Haptics-enabled expert-in-the-loop motor function development for such applications is made possible through multilateral telerobotic frameworks. While several studies have validated the benefits of haptic interaction with an expert in motor learning, contradictory results have also been reported. This emphasizes the need for further in-depth studies on the nature of human motor learning through haptic guidance and interaction. The objective of this study was to design and evaluate expert-in-the-loop multilateral telerobotic frameworks with stable and human-safe control loops that enable adaptive “hand-over-hand” haptic guidance for RAMRT and RAMIST. The first prerequisite for such frameworks is active involvement of the patient or trainee, which requires the closed-loop system to remain stable in the presence of an adaptable time-varying dominance factor. To this end, a wave-variable controller is proposed in this study for conventional trilateral teleoperation systems such that system stability is guaranteed in the presence of a time-varying dominance factor and communication delay. Similar to other wave-variable approaches, the controller is initially developed for the Velocity-force Domain (VD) based on the well-known passivity assumption on the human arm in VD. The controller can be applied straightforwardly to the Position-force Domain (PD), eliminating position-error accumulation and position drift, provided that passivity of the human arm in PD is addressed. However, the latter has been ignored in the literature. Therefore, in this study, passivity of the human arm in PD is investigated using mathematical analysis, experimentation as well as user studies involving 12 participants and 48 trials. The results, in conjunction with the proposed wave-variables, can be used to guarantee closed-loop PD stability of the supervised trilateral teleoperation system in its classical format. The classic dual-user teleoperation architecture does not, however, fully satisfy the requirements for properly imparting motor function (skills) in RAMRT (RAMIST). Consequently, the next part of this study focuses on designing novel supervised trilateral frameworks for providing motor learning in RAMRT and RAMIST, each customized according to the requirements of the application. The framework proposed for RAMRT includes the following features: a) therapist-in-the-loop mirror therapy; b) haptic feedback to the therapist from the patient side; c) assist-as-needed therapy realized through an adaptive Guidance Virtual Fixture (GVF); and d) real-time task-independent and patient-specific motor-function assessment. Closed-loop stability of the proposed framework is investigated using a combination of the Circle Criterion and the Small-Gain Theorem. The stability analysis addresses the instabilities caused by: a) communication delays between the therapist and the patient, facilitating haptics-enabled tele- or in-home rehabilitation; and b) the integration of the time-varying nonlinear GVF element into the delayed system. The platform is experimentally evaluated on a trilateral rehabilitation setup consisting of two Quanser rehabilitation robots and one Quanser HD2 robot. The framework proposed for RAMIST includes the following features: a) haptics-enabled expert-in-the-loop surgical training; b) adaptive expertise-oriented training, realized through a Fuzzy Interface System, which actively engages the trainees while providing them with appropriate skills-oriented levels of training; and c) task-independent skills assessment. Closed-loop stability of the architecture is analyzed using the Circle Criterion in the presence and absence of haptic feedback of tool-tissue interactions. In addition to the time-varying elements of the system, the stability analysis approach also addresses communication delays, facilitating tele-surgical training. The platform is implemented on a dual-console surgical setup consisting of the classic da Vinci surgical system (Intuitive Surgical, Inc., Sunnyvale, CA), integrated with the da Vinci Research Kit (dVRK) motor controllers, and the dV-Trainer master console (Mimic Technology Inc., Seattle, WA). In order to save on the expert\u27s (therapist\u27s) time, dual-console architectures can also be expanded to accommodate simultaneous training (rehabilitation) for multiple trainees (patients). As the first step in doing this, the last part of this thesis focuses on the development of a multi-master/single-slave telerobotic framework, along with controller design and closed-loop stability analysis in the presence of communication delays. Various parts of this study are supported with a number of experimental implementations and evaluations. The outcomes of this research include multilateral telerobotic testbeds for further studies on the nature of human motor learning and retention through haptic guidance and interaction. They also enable investigation of the impact of communication time delays on supervised haptics-enabled motor function improvement through tele-rehabilitation and mentoring

Trust-Based Control of (Semi)Autonomous Mobile Robotic Systems

Despite great achievements made in (semi)autonomous robotic systems, human participa-tion is still an essential part, especially for decision-making about the autonomy allocation of robots in complex and uncertain environments. However, human decisions may not be optimal due to limited cognitive capacities and subjective human factors. In human-robot interaction (HRI), trust is a major factor that determines humans use of autonomy. Over/under trust may lead to dispro-portionate autonomy allocation, resulting in decreased task performance and/or increased human workload. In this work, we develop automated decision-making aids utilizing computational trust models to help human operators achieve a more effective and unbiased allocation. Our proposed decision aids resemble the way that humans make an autonomy allocation decision, however, are unbiased and aim to reduce human workload, improve the overall performance, and result in higher acceptance by a human. We consider two types of autonomy control schemes for (semi)autonomous mobile robotic systems. The first type is a two-level control scheme which includes switches between either manual or autonomous control modes. For this type, we propose automated decision aids via a computational trust and self-confidence model. We provide analytical tools to investigate the steady-state effects of the proposed autonomy allocation scheme on robot performance and human workload. We also develop an autonomous decision pattern correction algorithm using a nonlinear model predictive control to help the human gradually adapt to a better allocation pattern. The second type is a mixed-initiative bilateral teleoperation control scheme which requires mixing of autonomous and manual control. For this type, we utilize computational two-way trust models. Here, mixed-initiative is enabled by scaling the manual and autonomous control inputs with a function of computational human-to-robot trust. The haptic force feedback cue sent by the robot is dynamically scaled with a function of computational robot-to-human trust to reduce humans physical workload. Using the proposed control schemes, our human-in-the-loop tests show that the trust-based automated decision aids generally improve the overall robot performance and reduce the operator workload compared to a manual allocation scheme. The proposed decision aids are also generally preferred and trusted by the participants. Finally, the trust-based control schemes are extended to the single-operator-multi-robot applications. A theoretical control framework is developed for these applications and the stability and convergence issues under the switching scheme between different robots are addressed via passivity based measures

Whole-Body Teleoperation and Shared Control of Redundant Robots with Applications to Aerial Manipulation

This paper introduces a passivity-based control framework for multi-task time-delayed bilateral teleoperation and shared control of kinematically-redundant robots. The proposed method can be seen as extension of state-of-the art hierarchical whole-body control as it allows for some of the tasks to be commanded by a remotely-located human operator through a haptic device while the others are autonomously performed. The operator is able to switch among tasks at any time without compromising the stability of the system. To enforce the passivity of the communication channel as well as to dissipate the energy generated by the null-space projectors used to enforce the hierarchy among the tasks, the Time-Domain Passivity Approach (TDPA) is applied. The efficacy of the approach is demonstrated through its application to the DLR Suspended Aerial Manipulator (SAM) in a real telemanipulation scenario with variable time delay, jitter, and package loss

A peer-to-peer Trilateral Passivity Control for delayed collaborative Teleoperation

In this paper a trilateral Multi-Master-Single-Slave-System with control authority allocation between two human operators is proposed. The authority coefficient permits to slide the dominant role between the operators. They can simultaneously execute a task in a collaborative way or a trainee might haptically only observe the task, while an expert is in full control. The master devices are connected with each other and the slave robot peer to peer without a central processing unit in a equitable way. The system design is general in that it allows delayed communication and different coupling causalities between masters and slave, which can be located far from each other. The Time Domain Passivity Control Approach guarantees passivity of the network in the presence of communication delays. The methods presented are sustained with simulations and experiments using different authority coefficients