Expert-in-the-Loop Multilateral Telerobotics for Haptics-Enabled Motor Function and Skills Development

Among medical robotics applications are Robotics-Assisted Mirror Rehabilitation Therapy (RAMRT) and Minimally-Invasive Surgical Training (RAMIST) that extensively rely on motor function development. Haptics-enabled expert-in-the-loop motor function development for such applications is made possible through multilateral telerobotic frameworks. While several studies have validated the benefits of haptic interaction with an expert in motor learning, contradictory results have also been reported. This emphasizes the need for further in-depth studies on the nature of human motor learning through haptic guidance and interaction. The objective of this study was to design and evaluate expert-in-the-loop multilateral telerobotic frameworks with stable and human-safe control loops that enable adaptive “hand-over-hand” haptic guidance for RAMRT and RAMIST. The first prerequisite for such frameworks is active involvement of the patient or trainee, which requires the closed-loop system to remain stable in the presence of an adaptable time-varying dominance factor. To this end, a wave-variable controller is proposed in this study for conventional trilateral teleoperation systems such that system stability is guaranteed in the presence of a time-varying dominance factor and communication delay. Similar to other wave-variable approaches, the controller is initially developed for the Velocity-force Domain (VD) based on the well-known passivity assumption on the human arm in VD. The controller can be applied straightforwardly to the Position-force Domain (PD), eliminating position-error accumulation and position drift, provided that passivity of the human arm in PD is addressed. However, the latter has been ignored in the literature. Therefore, in this study, passivity of the human arm in PD is investigated using mathematical analysis, experimentation as well as user studies involving 12 participants and 48 trials. The results, in conjunction with the proposed wave-variables, can be used to guarantee closed-loop PD stability of the supervised trilateral teleoperation system in its classical format. The classic dual-user teleoperation architecture does not, however, fully satisfy the requirements for properly imparting motor function (skills) in RAMRT (RAMIST). Consequently, the next part of this study focuses on designing novel supervised trilateral frameworks for providing motor learning in RAMRT and RAMIST, each customized according to the requirements of the application. The framework proposed for RAMRT includes the following features: a) therapist-in-the-loop mirror therapy; b) haptic feedback to the therapist from the patient side; c) assist-as-needed therapy realized through an adaptive Guidance Virtual Fixture (GVF); and d) real-time task-independent and patient-specific motor-function assessment. Closed-loop stability of the proposed framework is investigated using a combination of the Circle Criterion and the Small-Gain Theorem. The stability analysis addresses the instabilities caused by: a) communication delays between the therapist and the patient, facilitating haptics-enabled tele- or in-home rehabilitation; and b) the integration of the time-varying nonlinear GVF element into the delayed system. The platform is experimentally evaluated on a trilateral rehabilitation setup consisting of two Quanser rehabilitation robots and one Quanser HD2 robot. The framework proposed for RAMIST includes the following features: a) haptics-enabled expert-in-the-loop surgical training; b) adaptive expertise-oriented training, realized through a Fuzzy Interface System, which actively engages the trainees while providing them with appropriate skills-oriented levels of training; and c) task-independent skills assessment. Closed-loop stability of the architecture is analyzed using the Circle Criterion in the presence and absence of haptic feedback of tool-tissue interactions. In addition to the time-varying elements of the system, the stability analysis approach also addresses communication delays, facilitating tele-surgical training. The platform is implemented on a dual-console surgical setup consisting of the classic da Vinci surgical system (Intuitive Surgical, Inc., Sunnyvale, CA), integrated with the da Vinci Research Kit (dVRK) motor controllers, and the dV-Trainer master console (Mimic Technology Inc., Seattle, WA). In order to save on the expert\u27s (therapist\u27s) time, dual-console architectures can also be expanded to accommodate simultaneous training (rehabilitation) for multiple trainees (patients). As the first step in doing this, the last part of this thesis focuses on the development of a multi-master/single-slave telerobotic framework, along with controller design and closed-loop stability analysis in the presence of communication delays. Various parts of this study are supported with a number of experimental implementations and evaluations. The outcomes of this research include multilateral telerobotic testbeds for further studies on the nature of human motor learning and retention through haptic guidance and interaction. They also enable investigation of the impact of communication time delays on supervised haptics-enabled motor function improvement through tele-rehabilitation and mentoring

Advanced teleoperation and control system for industrial robots based on augmented virtuality and haptic feedback

There are some industrial tasks that are still mainly performed manually by human workers due to their complexity, which is the case of surface treatment operations (such as sanding, deburring, finishing, grinding, polishing, etc.) used to repair defects. This work develops an advanced teleoperation and control system for industrial robots in order to assist the human operator to perform the mentioned tasks. On the one hand, the controlled robotic system provides strength and accuracy, holding the tool, keeping the right tool orientation and guaranteeing a smooth approach to the workpiece. On the other hand, the advanced teleoperation provides security and comfort to the user when performing the task. In particular, the proposed teleoperation uses augmented virtuality (i.e., a virtual world that includes non-modeled real-world data) and haptic feedback to provide the user an immersive virtual experience when remotely teleoperating the tool of the robot system to treat arbitrary regions of the workpiece surface. The method is illustrated with a car body surface treatment operation, although it can be easily extended to other surface treatment applications or even to other industrial tasks where the human operator may benefit from robotic assistance. The effectiveness of the proposed approach is shown with several experiments using a 6R robotic arm. Moreover, a comparison of the performance obtained manually by an expert and that obtained with the proposed method has also been conducted in order to show the suitability of the proposed approach

Adaptive Neural Network Fixed-Time Control Design for Bilateral Teleoperation With Time Delay.

In this article, subject to time-varying delay and uncertainties in dynamics, we propose a novel adaptive fixed-time control strategy for a class of nonlinear bilateral teleoperation systems. First, an adaptive control scheme is applied to estimate the upper bound of delay, which can resolve the predicament that delay has significant impacts on the stability of bilateral teleoperation systems. Then, radial basis function neural networks (RBFNNs) are utilized for estimating uncertainties in bilateral teleoperation systems, including dynamics, operator, and environmental models. Novel adaptation laws are introduced to address systems' uncertainties in the fixed-time convergence settings. Next, a novel adaptive fixed-time neural network control scheme is proposed. Based on the Lyapunov stability theory, the bilateral teleoperation systems are proved to be stable in fixed time. Finally, simulations and experiments are presented to verify the validity of the control algorithm

Virtual Reality-Based Interface for Advanced Assisted Mobile Robot Teleoperation

[EN] This work proposes a new interface for the teleoperation of mobile robots based on virtual reality that allows a natural and intuitive interaction and cooperation between the human and the robot, which is useful for many situations, such as inspection tasks, the mapping of complex environments, etc. Contrary to previous works, the proposed interface does not seek the realism of the virtual environment but provides all the minimum necessary elements that allow the user to carry out the teleoperation task in a more natural and intuitive way. The teleoperation is carried out in such a way that the human user and the mobile robot cooperate in a synergistic way to properly accomplish the task: the user guides the robot through the environment in order to benefit from the intelligence and adaptability of the human, whereas the robot is able to automatically avoid collisions with the objects in the environment in order to benefit from its fast response. The latter is carried out using the well-known potential field-based navigation method. The efficacy of the proposed method is demonstrated through experimentation with the Turtlebot3 Burger mobile robot in both simulation and real-world scenarios. In addition, usability and presence questionnaires were also conducted with users of different ages and backgrounds to demonstrate the benefits of the proposed approach. In particular, the results of these questionnaires show that the proposed virtual reality based interface is intuitive, ergonomic and easy to use.This research was funded by the Spanish Government (Grant PID2020-117421RB-C21 funded byMCIN/AEI/10.13039/501100011033) and by the Generalitat Valenciana (Grant GV/2021/181).Solanes, JE.; Muñoz García, A.; Gracia Calandin, LI.; Tornero Montserrat, J. (2022). Virtual Reality-Based Interface for Advanced Assisted Mobile Robot Teleoperation. Applied Sciences. 12(12):1-22. https://doi.org/10.3390/app12126071122121

Bimanual robot control for surface treatment tasks

This work develops a method to perform surface treatment tasks using a bimanual robotic system, i.e. two robot arms cooperatively performing the task. In particular, one robot arm holds the workpiece while the other robot arm has the treatment tool attached to its end-effector. Moreover, the human user teleoperates all the six coordinates of the former robot arm and two coordinates of the latter robot arm, i.e. the teleoperator can move the treatment tool on the plane given by the workpiece surface. Furthermore, a force sensor attached to the treatment tool is used to automatically attain the desired pressure between the tool and the workpiece and to automatically keep the tool orientation orthogonal to the workpiece surface. In addition, to assist the human user during the teleoperation, several constraints are defined for both robot arms in order to avoid exceeding the allowed workspace, e.g. to avoid collisions with other objects in the environment. The theory used in this work to develop the bimanual robot control relies on sliding mode control and task prioritisation. Finally, the feasibility and effectiveness of the method are shown through experimental results using two robot arms

Bimanual robot control for surface treatment tasks

This is an Author's Accepted Manuscript of an article published in Alberto García, J. Ernesto Solanes, Luis Gracia, Pau Muñoz-Benavent, Vicent Girbés-Juan & Josep Tornero (2022) Bimanual robot control for surface treatment tasks, International Journal of Systems Science, 53:1, 74-107, DOI: 10.1080/00207721.2021.1938279 [copyright Taylor & Francis], available online at: http://www.tandfonline.com/10.1080/00207721.2021.1938279[EN] This work develops a method to perform surface treatment tasks using a bimanual robotic system, i.e. two robot arms cooperatively performing the task. In particular, one robot arm holds the work-piece while the other robot arm has the treatment tool attached to its end-effector. Moreover, the human user teleoperates all the six coordinates of the former robot arm and two coordinates of the latter robot arm, i.e. the teleoperator can move the treatment tool on the plane given by the work- piece surface. Furthermore, a force sensor attached to the treatment tool is used to automatically attain the desired pressure between the tool and the workpiece and to automatically keep the tool orientation orthogonal to the workpiece surface. In addition, to assist the human user during the teleoperation, several constraints are defined for both robot arms in order to avoid exceeding the allowed workspace, e.g. to avoid collisions with other objects in the environment. The theory used in this work to develop the bimanual robot control relies on sliding mode control and task prioritisation. Finally, the feasibility and effectiveness of the method are shown through experimental results using two robot arms.This work was supported by Generalitat Valenciana [grant numbers ACIF/2019/007 and GV/2021/181] and Spanish Ministry of Science and Innovation [grant number PID2020117421RB-C21].García-Fernández, A.; Solanes, JE.; Gracia Calandin, LI.; Muñoz-Benavent, P.; Girbés-Juan, V.; Tornero, J. (2022). Bimanual robot control for surface treatment tasks. International Journal of Systems Science. 53(1):74-107. https://doi.org/10.1080/00207721.2021.19382797410753

Modeling, Control, and Motion Analysis of a Class of Extensible Continuum Manipulators

In this dissertation, the development of a kinematic model, a configuration-space controller, a master-slave teleoperation controller, along with the analysis of the self-motion properties for redundant, extensible, continuous backbone (continuum) ``trunk and tentacle\u27 manipulators are detailed. Unlike conventional rigid-link robots, continuum manipulators are robots that can bend at any point along their backbone, resulting in new and unique modeling and control issues. Taken together, these chapters represent one of the first efforts towards devising model-based controllers of such robots, as well as characterizing their self-motion in its simplest form. Chapter 2 describes the development of a convenient set of generalized, spatial forward kinematics for extensible continuum manipulators based on the robot\u27s measurable variables. This development, takes advantage of the standard constant curvature assumption made for such manipulators and is simpler and more intuitive than the existing kinematic derivations which utilize a pseudo-rigid link manipulator. In Chapter 3, a new control strategy for continuum robots is presented. Control of this emerging new class of robots has proved difficult due to the inherent complexity of their dynamics. Using a recently established full Lagrangian dynamic model, a new nonlinear model-based control strategy (sliding-mode control) for continuum robots is introduced. Simulation results are illustrated using the dynamic model of a three-section, six Degree-of-Freedom, planar continuum robot and an experiment was conducted on the OctArm 9 Degree-of-Freedom continuum manipulator. In both the simulation and experiment, the results of the sliding-mode controller were found to be significantly better than a standard inverse-dynamics PD controller. In Chapter 4, the nature of continuum manipulator self-motion is studied. While use of the redundant continuum manipulator self-motion property (configuration changes which leave the end-effector location fixed) has been proposed, the nature of their null-spaces has not previously been explored. The manipulator related resolved-motion rate inverse kinematics which are based on the forward kinematics described in Chapter 2, are used. Based on these derivations, the self-motion of a 2-section, extensible redundant continuum manipulator in planar and spatial situations (generalizable to n-sections) is analyzed. The existence of a single self-motion manifold underlying the structures is proven, and simple self-motion cases spanning the null-space are introduced. The results of this analysis allow for a better understanding of general continuum robot self-motions and relate their underlying structure to real world examples and applications. The results are supported by experimental validation of the self-motion properties on the 9 Degree-of-Freedom OctArm continuum manipulator. In Chapter 5, teleoperation control of a kinematically redundant, continuum slave robot by a non-redundant, rigid-link master system is described. This problem is novel because the self-motion of the redundant robot can be utilized to achieve secondary control objectives while allowing the user to only control the tip of the slave system. To that end, feedback linearizing controllers are proposed for both the master and slave systems, whose effectiveness is demonstrated using numerical simulations and experimental results (using the 9 Degree-of-Freedom OctArm continuum manipulator as the slave system) for trajectory tracking as well as singularity avoidance subtask

STABILITY AND PERFORMANCE OF NETWORKED CONTROL SYSTEMS

Network control systems (NCSs), as one of the most active research areas, are arousing comprehensive concerns along with the rapid development of network. This dissertation mainly discusses the stability and performance of NCSs into the following two parts. In the first part, a new approach is proposed to reduce the data transmitted in networked control systems (NCSs) via model reduction method. Up to our best knowledge, we are the first to propose this new approach in the scientific and engineering society. The "unimportant" information of system states vector is truncated by balanced truncation method (BTM) before sending to the networked controller via network based on the balance property of the remote controlled plant controllability and observability. Then, the exponential stability condition of the truncated NCSs is derived via linear matrix inequality (LMI) forms. This method of data truncation can usually reduce the time delay and further improve the performance of the NCSs. In addition, all the above results are extended to the switched NCSs. The second part presents a new robust sliding mode control (SMC) method for general uncertain time-varying delay stochastic systems with structural uncertainties and the Brownian noise (Wiener process). The key features of the proposed method are to apply singular value decomposition (SVD) to all structural uncertainties, to introduce adjustable parameters for control design along with the SMC method, and new Lyapunov-type functional. Then, a less-conservative condition for robust stability and a new robust controller for the general uncertain stochastic systems are derived via linear matrix inequality (LMI) forms. The system states are able to reach the SMC switching surface as guaranteed in probability 1 by the proposed control rule. Furthermore, the novel Lyapunov-type functional for the uncertain stochastic systems is used to design a new robust control for the general case where the derivative of time-varying delay can be any bounded value (e.g., greater than one). It is theoretically proved that the conservatism of the proposed method is less than the previous methods. All theoretical proofs are presented in the dissertation. The simulations validate the correctness of the theoretical results and have better performance than the existing results

The Shape of Damping: Optimizing Damping Coefficients to Improve Transparency on Bilateral Telemanipulation

This thesis presents a novel optimization-based passivity control algorithm for hapticenabled bilateral teleoperation systems involving multiple degrees of freedom. In particular, in the context of energy-bounding control, the contribution focuses on the implementation of a passivity layer for an existing time-domain scheme, ensuring optimal transparency of the interaction along subsets of the environment space which are preponderant for the given task, while preserving the energy bounds required for passivity. The involved optimization problem is convex and amenable to real-time implementation. The effectiveness of the proposed design is validated via an experiment performed on a virtual teleoperated environment. The interplay between transparency and stability is a critical aspect in haptic-enabled bilateral teleoperation control. While it is important to present the user with the true impedance of the environment, destabilizing factors such as time delays, stiff environments, and a relaxed grasp on the master device may compromise the stability and safety of the system. Passivity has been exploited as one of the the main tools for providing sufficient conditions for stable teleoperation in several controller design approaches, such as the scattering algorithm, timedomain passivity control, energy bounding algorithm, and passive set position modulation. In this work it is presented an innovative energy-based approach, which builds upon existing time-domain passivity controllers, improving and extending their effectiveness and functionality. The set of damping coefficients are prioritized in each degree of freedom, the resulting transparency presents a realistic force feedback in comparison to the other directions. Thus, the prioritization takes effect using a quadratic programming algorithm to find the optimal values for the damping. Finally, the energy tanks approach on passivity control is a solution used to ensure stability in a system for robotics bilateral manipulation. The bilateral telemanipulation must maintain the principle of passivity in all moments to preserve the system\u2019s stability. This work presents a brief introduction to haptic devices as a master component on the telemanipulation chain; the end effector in the slave side is a representation of an interactive object within an environment having a force sensor as feedback signal. The whole interface is designed into a cross-platform framework named ROS, where the user interacts with the system. Experimental results are presented