Expert-in-the-Loop Multilateral Telerobotics for Haptics-Enabled Motor Function and Skills Development

Among medical robotics applications are Robotics-Assisted Mirror Rehabilitation Therapy (RAMRT) and Minimally-Invasive Surgical Training (RAMIST) that extensively rely on motor function development. Haptics-enabled expert-in-the-loop motor function development for such applications is made possible through multilateral telerobotic frameworks. While several studies have validated the benefits of haptic interaction with an expert in motor learning, contradictory results have also been reported. This emphasizes the need for further in-depth studies on the nature of human motor learning through haptic guidance and interaction. The objective of this study was to design and evaluate expert-in-the-loop multilateral telerobotic frameworks with stable and human-safe control loops that enable adaptive “hand-over-hand” haptic guidance for RAMRT and RAMIST. The first prerequisite for such frameworks is active involvement of the patient or trainee, which requires the closed-loop system to remain stable in the presence of an adaptable time-varying dominance factor. To this end, a wave-variable controller is proposed in this study for conventional trilateral teleoperation systems such that system stability is guaranteed in the presence of a time-varying dominance factor and communication delay. Similar to other wave-variable approaches, the controller is initially developed for the Velocity-force Domain (VD) based on the well-known passivity assumption on the human arm in VD. The controller can be applied straightforwardly to the Position-force Domain (PD), eliminating position-error accumulation and position drift, provided that passivity of the human arm in PD is addressed. However, the latter has been ignored in the literature. Therefore, in this study, passivity of the human arm in PD is investigated using mathematical analysis, experimentation as well as user studies involving 12 participants and 48 trials. The results, in conjunction with the proposed wave-variables, can be used to guarantee closed-loop PD stability of the supervised trilateral teleoperation system in its classical format. The classic dual-user teleoperation architecture does not, however, fully satisfy the requirements for properly imparting motor function (skills) in RAMRT (RAMIST). Consequently, the next part of this study focuses on designing novel supervised trilateral frameworks for providing motor learning in RAMRT and RAMIST, each customized according to the requirements of the application. The framework proposed for RAMRT includes the following features: a) therapist-in-the-loop mirror therapy; b) haptic feedback to the therapist from the patient side; c) assist-as-needed therapy realized through an adaptive Guidance Virtual Fixture (GVF); and d) real-time task-independent and patient-specific motor-function assessment. Closed-loop stability of the proposed framework is investigated using a combination of the Circle Criterion and the Small-Gain Theorem. The stability analysis addresses the instabilities caused by: a) communication delays between the therapist and the patient, facilitating haptics-enabled tele- or in-home rehabilitation; and b) the integration of the time-varying nonlinear GVF element into the delayed system. The platform is experimentally evaluated on a trilateral rehabilitation setup consisting of two Quanser rehabilitation robots and one Quanser HD2 robot. The framework proposed for RAMIST includes the following features: a) haptics-enabled expert-in-the-loop surgical training; b) adaptive expertise-oriented training, realized through a Fuzzy Interface System, which actively engages the trainees while providing them with appropriate skills-oriented levels of training; and c) task-independent skills assessment. Closed-loop stability of the architecture is analyzed using the Circle Criterion in the presence and absence of haptic feedback of tool-tissue interactions. In addition to the time-varying elements of the system, the stability analysis approach also addresses communication delays, facilitating tele-surgical training. The platform is implemented on a dual-console surgical setup consisting of the classic da Vinci surgical system (Intuitive Surgical, Inc., Sunnyvale, CA), integrated with the da Vinci Research Kit (dVRK) motor controllers, and the dV-Trainer master console (Mimic Technology Inc., Seattle, WA). In order to save on the expert\u27s (therapist\u27s) time, dual-console architectures can also be expanded to accommodate simultaneous training (rehabilitation) for multiple trainees (patients). As the first step in doing this, the last part of this thesis focuses on the development of a multi-master/single-slave telerobotic framework, along with controller design and closed-loop stability analysis in the presence of communication delays. Various parts of this study are supported with a number of experimental implementations and evaluations. The outcomes of this research include multilateral telerobotic testbeds for further studies on the nature of human motor learning and retention through haptic guidance and interaction. They also enable investigation of the impact of communication time delays on supervised haptics-enabled motor function improvement through tele-rehabilitation and mentoring

Safe Haptics-enabled Patient-Robot Interaction for Robotic and Telerobotic Rehabilitation of Neuromuscular Disorders: Control Design and Analysis

Motivation: Current statistics show that the population of seniors and the incidence rate of age-related neuromuscular disorders are rapidly increasing worldwide. Improving medical care is likely to increase the survival rate but will result in even more patients in need of Assistive, Rehabilitation and Assessment (ARA) services for extended periods which will place a significant burden on the world\u27s healthcare systems. In many cases, the only alternative is limited and often delayed outpatient therapy. The situation will be worse for patients in remote areas. One potential solution is to develop technologies that provide efficient and safe means of in-hospital and in-home kinesthetic rehabilitation. In this regard, Haptics-enabled Interactive Robotic Neurorehabilitation (HIRN) systems have been developed. Existing Challenges: Although there are specific advantages with the use of HIRN technologies, there still exist several technical and control challenges, e.g., (a) absence of direct interactive physical interaction between therapists and patients; (b) questionable adaptability and flexibility considering the sensorimotor needs of patients; (c) limited accessibility in remote areas; and (d) guaranteeing patient-robot interaction safety while maximizing system transparency, especially when high control effort is needed for severely disabled patients, when the robot is to be used in a patient\u27s home or when the patient experiences involuntary movements. These challenges have provided the motivation for this research. Research Statement: In this project, a novel haptics-enabled telerobotic rehabilitation framework is designed, analyzed and implemented that can be used as a new paradigm for delivering motor therapy which gives therapists direct kinesthetic supervision over the robotic rehabilitation procedure. The system also allows for kinesthetic remote and ultimately in-home rehabilitation. To guarantee interaction safety while maximizing the performance of the system, a new framework for designing stabilizing controllers is developed initially based on small-gain theory and then completed using strong passivity theory. The proposed control framework takes into account knowledge about the variable biomechanical capabilities of the patient\u27s limb(s) in absorbing interaction forces and mechanical energy. The technique is generalized for use for classical rehabilitation robotic systems to realize patient-robot interaction safety while enhancing performance. In the next step, the proposed telerobotic system is studied as a modality of training for classical HIRN systems. The goal is to first model and then regenerate the prescribed kinesthetic supervision of an expert therapist. To broaden the population of patients who can use the technology and HIRN systems, a new control strategy is designed for patients experiencing involuntary movements. As the last step, the outcomes of the proposed theoretical and technological developments are translated to designing assistive mechatronic tools for patients with force and motion control deficits. This study shows that proper augmentation of haptic inputs can not only enhance the transparency and safety of robotic and telerobotic rehabilitation systems, but it can also assist patients with force and motion control deficiencies

A modular telerehabilitation architecture for upper limb robotic therapy

Several factors may prevent post-stroke subjects from participating in rehabilitation protocols, for example, geographical location of rehabilitation centres, socioeconomic status, economic burden and lack of logistics surrounding transportation. Early supported discharge from hospitals with continued rehabilitation at home represents a well-defined regimen of post-stroke treatment. Information-based technologies coupled with robotics have promoted the development of new technologies for telerehabilitation. In this article, the design and development of a modular architecture for delivering upper limb robotic telerehabilitation with the CBM-Motus, a planar unilateral robotic machine that allows performing state-of-the-art rehabilitation tasks, have been presented. The proposed architecture allows a therapist to set a therapy session on his or her side and send it to the patient's side with a standardized communication protocol; the user interacts with the robot that provides an adaptive assistance during the rehabilitation tasks. Patient's performance is evaluated by means of performance indicators, which are also used to update robot behaviour during assistance. The implementation of the architecture is described and a set of validation tests on seven healthy subjects are presented. Results show the reliability of the novel architecture and the capability to be easily tailored to the user's needs with the chosen robotic device

Development and evaluation of a haptic framework supporting telerehabilitation robotics and group interaction

Telerehabilitation robotics has grown remarkably in the past few years. It can provide intensive training to people with special needs remotely while facilitating therapists to observe the whole process. Telerehabilitation robotics is a promising solution supporting routine care which can help to transform face-to-face and one-on-one treatment sessions that require not only intensive human resource but are also restricted to some specialised care centres to treatments that are technology-based (less human involvement) and easy to access remotely from anywhere. However, there are some limitations such as network latency, jitter, and delay of the internet that can affect negatively user experience and quality of the treatment session. Moreover, the lack of social interaction since all treatments are performed over the internet can reduce motivation of the patients. As a result, these limitations are making it very difficult to deliver an efficient recovery plan. This thesis developed and evaluated a new framework designed to facilitate telerehabilitation robotics. The framework integrates multiple cutting-edge technologies to generate playful activities that involve group interaction with binaural audio, visual, and haptic feedback with robot interaction in a variety of environments. The research questions asked were: 1) Can activity mediated by technology motivate and influence the behaviour of users, so that they engage in the activity and sustain a good level of motivation? 2) Will working as a group enhance users’ motivation and interaction? 3) Can we transfer real life activity involving group interaction to virtual domain and deliver it reliably via the internet? There were three goals in this work: first was to compare people’s behaviours and motivations while doing the task in a group and on their own; second was to determine whether group interaction in virtual and reala environments was different from each other in terms of performance, engagement and strategy to complete the task; finally was to test out the effectiveness of the framework based on the benchmarks generated from socially assistive robotics literature. Three studies have been conducted to achieve the first goal, two with healthy participants and one with seven autistic children. The first study observed how people react in a challenging group task while the other two studies compared group and individual interactions. The results obtained from these studies showed that the group interactions were more enjoyable than individual interactions and most likely had more positive effects in terms of user behaviours. This suggests that the group interaction approach has the potential to motivate individuals to make more movements and be more active and could be applied in the future for more serious therapy. Another study has been conducted to measure group interaction’s performance in virtual and real environments and pointed out which aspect influences users’ strategy for dealing with the task. The results from this study helped to form a better understanding to predict a user’s behaviour in a collaborative task. A simulation has been run to compare the results generated from the predictor and the real data. It has shown that, with an appropriate training method, the predictor can perform very well. This thesis has demonstrated the feasibility of group interaction via the internet using robotic technology which could be beneficial for people who require social interaction (e.g. stroke patients and autistic children) in their treatments without regular visits to the clinical centres

Technology-supported training of arm-hand skills in stroke

Impaired arm-hand performance is a serious consequence of stroke that is associated with reduced self-efficacy and poor quality of life. Task-oriented arm training is a therapy approach that is known to improve skilled arm-hand performance, even in chronic stages after stroke. At the start of this project, little knowledge had been consolidated regarding taskoriented arm training characteristics, especially in the field of technology-supported rehabilitation. The feasibility and effects of technology-supported client-centred task-oriented training on skilled arm-hand performance had not been investigated but to a very limited degree. Reviewing literature on rehabilitation and motor learning in stroke led to the identification of therapy oriented criteria for rehabilitation technology aiming to influence skilled arm-hand performance (chapter 2). Most rehabilitation systems reported in literature to date are robotic systems that are aimed at providing an engaging exercise environment and feedback on motor performance. Both, feedback and engaging exercises are important for motivating patients to perform a high number of exercise repetitions and prolonged training, which are important factors for motor learning. The review also found that current rehabilitation technology is focussed mainly on providing treatment at a function level, thereby improving joint range of motion, muscle strength and parameters such as movement speed and smoothness of movement during analytical movements. However, related research has found no effects of robot-supported training at the activity level. The review concluded that a challenge exists for upper extremity rehabilitation technology in stroke patients to also provide more patienttailored task-oriented arm-hand training in natural environments to support the learning of skilled arm-hand performance. Besides mapping the strengths of different technological solutions, the use of outcome measures and training protocols needs to become more standardized across similar interventions, in order to help determine which training solutions are most suitable for specific patient categories. Chapter 4 contributes towards such a standardization of outcome measurement. A concept is introduced which may guide the clinician/researcher to choose outcome measures for evaluating specific and generalized training effects. As an initial operationalization of this concept, 28 test batteries that have been used in 16 task-oriented training interventions were rated as to whether measurement components were measured by the test. Future research is suggested that elaborates the concept with information on the relative weighing of components in each test, with more test batteries (which may lead to additional components) and by adding more test properties into the concept (e.g. psychometric properties of the tests, possible floor- or ceiling effects). Task-oriented training is one of the training approaches that has been shown to be beneficial for skilled arm-hand performance after stroke. Important mechanisms for motor learning that are identified are patient motivation for such training, and the learning of efficient goaloriented movement strategies and task-specific problem solving. In this thesis we operationalize task-oriented training in terms of 15 components (chapter 3). A systematic review that included 16 randomized controlled trials using task-oriented training in stroke patients, evaluated the effects of these training components on skilled arm-hand performance. The number of training components used in an intervention aimed at improving arm-hand performance after stroke was not associated with the post-treatment effect size. Distributed practice and feedback were associated with the largest post-intervention effect sizes. Random practice and use of clear functional training goals were associated with the largest follow-up effect sizes. It may be that training components that optimize the storage of learned motor performance in the long-term memory are associated with larger treatment effects. Unfortunately, feedback, random practice and distributed practice were reported in very few of the included randomized controlled trials (in only 6,3 and 1 out of the 17 studies respectively). Client-centred training, i.e. training on exercises that support goals that are selected by the patients themselves, improves patient motivation for training. Motivation in turn has proven to positively influence motor learning in stroke patients, as attention during training is heightened and storage of information in the long-term memory improves. Chapter 5 reports on an interview of 40 stroke patients, investigating into training preferences. A list of 46 skills, ranked according to descending training preference scores, was provided that can be used for implementation of exercises in rehabilitation technology, in order for technologysupported training to be client-centred. Chapter 6 introduces T-TOAT, a technology supported task-oriented arm training method that was developed together with colleagues at Adelante (Hoensbroek, NL). T-TOAT enables the implementation of exercises that support task-oriented training in rehabilitation technology. The training method is applicable for different technological systems, e.g. robot and sensor systems, or in combination with functional electrical stimulation, etc. To enable the use of TTOAT for training with the Haptic Master Robot (MOOG-FCS, NL), special software named Haptic TOAT was developed in Adelante together with colleagues at the Centre of Technology in Care of Zuyd University (chapter 6). The software enables the recording of the patient’s movement trajectories, given task constraints and patient possibilities, using the Haptic Master as a recording device. A purpose-made gimbal was attached to the endeffector, leaving the hand free for the use and manipulating objects. The recorded movement can be replayed in a passive mode or in an active mode (active, active-assisted or activeresisted). Haptic feedback is provided when the patient deviates from the recorded movement trajectory, as the patient receives the sensation of bouncing into a wall, as well as feeling a spring that pulls him/her back to the recorded path. The diameter of the tunnel around the recorded trajectory (distance to the wall), and the spring force can be adjusted for each patient. An ongoing clinical trial in which chronic stroke patients train with Haptic-TOAT examines whether Haptic Master provides additional value compared to supporting the same exercises by video-instruction only. Together with Philips Research Europe (Eindhoven,Aachen), the T-TOAT method has been implemented in a sensor based prototype, called Philips Stroke Rehabilitation Exerciser. This system included movement tracking sensors and an exercise board interacting with real life objects. A very strong feature of the system is that feedback is provided to patients (real-time and after exercise performance), based on a comparison of the patient’s exercise performance to individual targets set by the therapist. Chapter 7 reports on a clinical trial investigating arm-hand treatment outcome and patient motivation for technology-supported task-oriented training in chronic stroke patients. It was found that 8 weeks of T-TOAT training improved arm-hand performance in chronic stroke patients significantly on Fugl-Meyer, Action Research Arm Test, and Motor Activity Log. An improvement was found in health-related quality of life. Training effects lasted at least 6 months post-training. Participants reported feeling intrinsically motivated and competent to use the system. The results of this study showed that T-TOAT is feasible. Despite the small number of stroke patients tested (n=9), significant and clinically relevant improvements in skilled arm-hand performance were found. In conclusion, this thesis has made several contributions. It motivated the need for clientcentred task-oriented training, which it has operationalized in terms of 15 components. Four of these 15 components were identified as most beneficial for the patient. A prioritized inventory of arm-hand training preferences of stroke patients was compiled by means of an interview study of 40 subacute and chronic stroke patients. T-TOAT, a method for technology-supported, client-centred, task-oriented training, was conceived and implemented in two target technologies (Haptic Master and Philips Stroke Rehabilitation Exerciser). Its feasibility was demonstrated in a clinical trial showing substantial and durable benefits for the stroke patients. Finally, the thesis contributes towards the standardization of outcome measures which is necessary for charting progress and guiding future developments of technology-supported stroke rehabilitation. Methodological considerations were discussed and several suggestions for future research were presented. The variety of treatment approaches and the various ways of support and challenge that are offered by existing rehabilitation technologies hold a large potential for offering a variety of extra training opportunities to stroke patients that may improve their arm-hand performance. Such solutions will be of increasing importance, to alleviate therapists and reduce economic pressure on the health care system, as the stroke incidence is increasing rapidly over the coming decades

A Framework for Prediction in a Fog-Based Tactile Internet Architecture for Remote Phobia Treatment

Tactile Internet, as the next generation of the Internet, aims to transmit the modality of touch in addition to conventional audiovisual signals, thus transforming today’s content-delivery into skill-set delivery networks, which promises ultra-low latency and ultra reliability. Besides voice and data communication driving the design of the current Internet, Tactile Internet enables haptic communications by incorporating 5G networks and edge computing. A novel use-case of immersive, low-latency Tactile Internet applications is haptic-enabled Virtual Reality (VR), where an extremely low latency of less than 50 ms is required, which gives way to the so-called Remote Phobia treatment via VR. It is a greenfield in the telehealth domain with the goal of replicating normal therapy sessions with distant therapists and patients, thereby standing as a cost-efficient and time-saving solution. In this thesis, we consider a recently proposed fog-based haptic-enabled VR system for remote treatment of animal phobia consisting of three main components: (1) therapist-side fog domain, (2) core network, and (3) patient-side fog domain. The patient and therapist domains are located in different fog domains, where their communication takes place through the core network. The therapist tries to cure the phobic patient remotely via a shared haptic virtual reality environment. However, certain haptic sensation messages associated with hand movements might not be reached in time, even in the most reliable networks. In this thesis, a prediction model is proposed to address the problem of excessive packet latency as well as packet loss, which may result in quality-of-experience (QoE) degradation. We aim to use machine learning to decouple the impact of excessive latency and extreme packet loss from the user experience perspective. For which, we propose a predictive framework called Edge Tactile Learner (ETL). Our proposed fog-based framework is responsible for predicting the zones touched by the therapist’s hand, then delivering it immediately to the patient-side fog domain if needed. The proposed ETL builds a model based on Weighted K-Nearest Neighbors (WKNN) to predict the zones touched by the therapist in a VR phobia treatment system. The simulation results indicate that our proposed predictive framework is instrumental in providing accurate and real-time haptic predictions to the patient-side fog domain. This increases patient’s immersion and synchronization between multiple senses such as audio, visual and haptic sensory, which leads to higher user Quality of Experience (QoE)

Technology-assisted training of arm-hand skills in stroke: concepts on reacquisition of motor control and therapist guidelines for rehabilitation technology design

<p>Abstract</p> <p>Background</p> <p>It is the purpose of this article to identify and review criteria that rehabilitation technology should meet in order to offer arm-hand training to stroke patients, based on recent principles of motor learning.</p> <p>Methods</p> <p>A literature search was conducted in PubMed, MEDLINE, CINAHL, and EMBASE (1997–2007).</p> <p>Results</p> <p>One hundred and eighty seven scientific papers/book references were identified as being relevant. Rehabilitation approaches for upper limb training after stroke show to have shifted in the last decade from being analytical towards being focussed on environmentally contextual skill training (task-oriented training). Training programmes for enhancing motor skills use patient and goal-tailored exercise schedules and individual feedback on exercise performance. Therapist criteria for upper limb rehabilitation technology are suggested which are used to evaluate the strengths and weaknesses of a number of current technological systems.</p> <p>Conclusion</p> <p>This review shows that technology for supporting upper limb training after stroke needs to align with the evolution in rehabilitation training approaches of the last decade. A major challenge for related technological developments is to provide engaging patient-tailored task oriented arm-hand training in natural environments with patient-tailored feedback to support (re) learning of motor skills.</p

A review on manipulation skill acquisition through teleoperation-based learning from demonstration

Manipulation skill learning and generalization have gained increasing attention due to the wide applications of robot manipulators and the spurt of robot learning techniques. Especially, the learning from demonstration method has been exploited widely and successfully in the robotic community, and it is regarded as a promising direction to realize the manipulation skill learning and generalization. In addition to the learning techniques, the immersive teleoperation enables the human to operate a remote robot with an intuitive interface and achieve the telepresence. Thus, it is a promising way to transfer manipulation skills from humans to robots by combining the learning methods and the teleoperation, and adapting the learned skills to different tasks in new situations. This review, therefore, aims to provide an overview of immersive teleoperation for skill learning and generalization to deal with complex manipulation tasks. To this end, the key technologies, e.g. manipulation skill learning, multimodal interfacing for teleoperation and telerobotic control, are introduced. Then, an overview is given in terms of the most important applications of immersive teleoperation platform for robot skill learning. Finally, this survey discusses the remaining open challenges and promising research topics

HAPTICS IN ROBOTICS AND AUTOMOTIVE SYSTEMS

Haptics is the science of applying touch (tactile) sensation and control to interaction with computer applications. The devices used to interact with computer applications are known as haptic interfaces. These devices sense some form of human movement, be it finger, head, hand or body movement and receive feedback from computer applications in form of felt sensations to the limbs or other parts of the human body. Examples of haptic interfaces range from force feedback joysticks/controllers in video game consoles to tele-operative surgery. This thesis deals with haptic interfaces involving hand movements. The first experiment involves using the end effector of a robotic manipulator as an interactive device to aid patients with deficits in the upper extremities in passive resistance therapy using novel path planning. The second experiment involves the application of haptic technology to the human-vehicle interface in a steer-by-wire transportation system using adaptive control