A Framework for Predicting Haptic Feedback in Needle Insertion in 5G Remote Robotic Surgery

© 2020 IEEE. Robots are being used more and more in surgery due to the many benefits they bring (e.g. reduction of patient discomfort, precision, reliability). Remote robotic surgery is now expected to become a reality due to the emergence of 5G. Needle insertion is a crucial element of many robotic surgical procedures such as biopsies, injections, neurosurgery, and brachytherapy cancer treatment. During needle insertion in remote robotic surgery, there is still no guarantee that the surgeon will obtain the haptic feedback from the patient side within the stringent deadlines, even in 5G settings. This paper proposes a framework for learning by imitation as a way to predict the messages that will eventually fail to reach their destination within the required deadlines. By leveraging expert demonstrations, the Hidden Markov Model is used to encapsulate a set of expert force/torque profiles and corresponding parameters during the off-line training process. A Gaussian mixture regression is then used to reproduce a generalized version of the force/torque profile and corresponding parameters during the prediction. Simulations are conducted to evaluate the performance of the proposed method. They show that our proposed framework is able to execute predictions in much less than the 1ms end-to-end latency requirement of remote robotic surgery

Recent Advancements in Augmented Reality for Robotic Applications: A Survey

Robots are expanding from industrial applications to daily life, in areas such as medical robotics, rehabilitative robotics, social robotics, and mobile/aerial robotics systems. In recent years, augmented reality (AR) has been integrated into many robotic applications, including medical, industrial, human–robot interactions, and collaboration scenarios. In this work, AR for both medical and industrial robot applications is reviewed and summarized. For medical robot applications, we investigated the integration of AR in (1) preoperative and surgical task planning; (2) image-guided robotic surgery; (3) surgical training and simulation; and (4) telesurgery. AR for industrial scenarios is reviewed in (1) human–robot interactions and collaborations; (2) path planning and task allocation; (3) training and simulation; and (4) teleoperation control/assistance. In addition, the limitations and challenges are discussed. Overall, this article serves as a valuable resource for working in the field of AR and robotic research, offering insights into the recent state of the art and prospects for improvement

A Comprehensive Survey of the Tactile Internet: State of the art and Research Directions

The Internet has made several giant leaps over the years, from a fixed to a mobile Internet, then to the Internet of Things, and now to a Tactile Internet. The Tactile Internet goes far beyond data, audio and video delivery over fixed and mobile networks, and even beyond allowing communication and collaboration among things. It is expected to enable haptic communication and allow skill set delivery over networks. Some examples of potential applications are tele-surgery, vehicle fleets, augmented reality and industrial process automation. Several papers already cover many of the Tactile Internet-related concepts and technologies, such as haptic codecs, applications, and supporting technologies. However, none of them offers a comprehensive survey of the Tactile Internet, including its architectures and algorithms. Furthermore, none of them provides a systematic and critical review of the existing solutions. To address these lacunae, we provide a comprehensive survey of the architectures and algorithms proposed to date for the Tactile Internet. In addition, we critically review them using a well-defined set of requirements and discuss some of the lessons learned as well as the most promising research directions

Task-oriented joint design of communication and computing for Internet of Skills

Nowadays, the internet is taking a revolutionary step forward, which is known as Internet of Skills. The Internet of Skills is a concept that refers to a network of sensors, actuators, and machines that enable knowledge, skills, and expertise delivery between people and machines, regardless of their geographical locations. This concept allows an immersive remote operation and access to expertise through virtual and augmented reality, haptic communications, robotics, and other cutting-edge technologies with various applications, including remote surgery and diagnosis in healthcare, remote laboratory and training in education, remote driving in transportation, and advanced manufacturing in Industry 4.0. In this thesis, we investigate three fundamental communication requirements of Internet of Skills applications, namely ultra-low latency, ultra-high reliability, and wireless resource utilization efficiency. Although 5G communications provide cutting-edge solutions for achieving ultra-low latency and ultra-high reliability with good resource utilization efficiency, meeting these requirements is difficult, particularly in long-distance communications where the distance between source and destination is more than 300 km, considering delays and reliability issues in networking components as well as physical limits of the speed of light. Furthermore, resource utilization efficiency must be improved further to accommodate the rapidly increasing number of mobile devices. Therefore, new design techniques that take into account both communication and computing systems with the task-oriented approach are urgently needed to satisfy conflicting latency and reliability requirements while improving resource utilization efficiency. First, we design and implement a 5G-based teleoperation prototype for Internet of Skills applications. We presented two emerging Internet of Skills use cases in healthcare and education. We conducted extensive experiments evaluating local and long-distance communication latency and reliability to gain insights into the current capabilities and limitations. From our local experiments in laboratory environment where both operator and robot in the same room, we observed that communication latency is around 15 ms with a 99.9% packet reception rate (communication reliability). However, communication latency increases up to 2 seconds in long-distance scenarios (between the UK and China), while it is around 50-300 ms within the UK experiments. In addition, our observations revealed that communication reliability and overall system performance do not exhibit a direct correlation. Instead, the number of consecutive packet drops emerged as the decisive factor influencing the overall system performance and user quality of experience. In light of these findings, we proposed a two-way timeout approach. We discarded stale packets to mitigate waiting times effectively and, in turn, reduce the latency. Nevertheless, we observed that the proposed approach reduced latency at the expense of reliability, thus verifying the challenge of the conflicting latency and reliability requirements. Next, we propose a task-oriented prediction and communication co-design framework to meet conflicting latency and reliability requirements. The proposed framework demonstrates the task-oriented joint design of communication and computing systems, where we considered packet losses in communications and prediction errors in prediction algorithms to derive the upper bound for overall system reliability. We revealed the tradeoff between overall system reliability and resource utilization efficiency, where we consider 5G NR as an example communication system. The proposed framework is evaluated with real-data samples and generated synthetic data samples. From the results, the proposed framework achieves better latency and reliability tradeoff with a 77.80% resource utilization efficiency improvement compared to a task-agnostic benchmark. In addition, we demonstrate that deploying a predictor at the receiver side achieves better overall reliability compared to a system that predictor at the transmitter. Finally, we propose an intelligent mode-switching framework to address the resource utilization challenge. We jointly design the communication, user intention recognition, and modeswitching systems to reduce communication load subject to joint task completion probability. We reveal the tradeoff between task prediction accuracy and task observation length, showing that higher prediction accuracy can be achieved when the task observation length increases. The proposed framework achieves more than 90% task prediction accuracy with 60% observation length. We train a DRL agent with real-world data from our teleoperation prototype for modeswitching between teleoperation and autonomous modes. Our results show that the proposed framework achieves up to 50% communication load reduction with similar task completion probability compared to conventional teleoperation

Task-oriented prediction and communication co-design for haptic communications

Prediction has recently been considered as a promising approach to meet low-latency and high-reliability requirements in long-distance haptic communications. However, most of the existing methods did not take features of tasks and the relationship between prediction and communication into account. In this paper, we propose a task-oriented prediction and communication co-design framework, where the reliability of the system depends on prediction errors and packet losses in communications. The goal is to minimize the required radio resources subject to the low-latency and high-reliability requirements of various tasks. Specifically, we consider the just noticeable difference (JND) as a performance metric for the haptic communication system. We collect experiment data from a real-world teleoperation testbed and use time-series generative adversarial networks (TimeGAN) to generate a large amount of synthetic data. This allows us to obtain the relationship between the JND threshold, prediction horizon, and the overall reliability including communication reliability and prediction reliability. We take 5G New Radio as an example to demonstrate the proposed framework and optimize bandwidth allocation and data rates of devices. Our numerical and experimental results show that the proposed framework can reduce wireless resource consumption up to 77.80% compared with a task-agnostic benchmark

A Framework for Prediction in a Fog-Based Tactile Internet Architecture for Remote Phobia Treatment

Tactile Internet, as the next generation of the Internet, aims to transmit the modality of touch in addition to conventional audiovisual signals, thus transforming today’s content-delivery into skill-set delivery networks, which promises ultra-low latency and ultra reliability. Besides voice and data communication driving the design of the current Internet, Tactile Internet enables haptic communications by incorporating 5G networks and edge computing. A novel use-case of immersive, low-latency Tactile Internet applications is haptic-enabled Virtual Reality (VR), where an extremely low latency of less than 50 ms is required, which gives way to the so-called Remote Phobia treatment via VR. It is a greenfield in the telehealth domain with the goal of replicating normal therapy sessions with distant therapists and patients, thereby standing as a cost-efficient and time-saving solution. In this thesis, we consider a recently proposed fog-based haptic-enabled VR system for remote treatment of animal phobia consisting of three main components: (1) therapist-side fog domain, (2) core network, and (3) patient-side fog domain. The patient and therapist domains are located in different fog domains, where their communication takes place through the core network. The therapist tries to cure the phobic patient remotely via a shared haptic virtual reality environment. However, certain haptic sensation messages associated with hand movements might not be reached in time, even in the most reliable networks. In this thesis, a prediction model is proposed to address the problem of excessive packet latency as well as packet loss, which may result in quality-of-experience (QoE) degradation. We aim to use machine learning to decouple the impact of excessive latency and extreme packet loss from the user experience perspective. For which, we propose a predictive framework called Edge Tactile Learner (ETL). Our proposed fog-based framework is responsible for predicting the zones touched by the therapist’s hand, then delivering it immediately to the patient-side fog domain if needed. The proposed ETL builds a model based on Weighted K-Nearest Neighbors (WKNN) to predict the zones touched by the therapist in a VR phobia treatment system. The simulation results indicate that our proposed predictive framework is instrumental in providing accurate and real-time haptic predictions to the patient-side fog domain. This increases patient’s immersion and synchronization between multiple senses such as audio, visual and haptic sensory, which leads to higher user Quality of Experience (QoE)

Modeling and Force Estimation of Cardiac Catheters for Haptics-enabled Tele-intervention

Robot-assisted cardiovascular intervention (RCI) systems have shown success in reducing the x-ray exposure to surgeons and patients during cardiovascular interventional procedures. RCI systems typically are teleoperated systems with leader-follower architecture. With such system architecture, the surgeon is placed out of the x-ray exposure zone and uses a console to control the robot remotely. Despite its success in reducing x-ray exposure, clinicians have identified the lack of force feedback as to its main technological limitation that can lead to vascular perforation of the patient’s vessels and even their death. The objective of this thesis was to develop, verify, and validate mechatronics technology for real-time accurate and robust haptic feedback rendering for RCI systems. To attain the thesis objective, first, a thorough review of the state-of-the-art clinical requirements, modeling approaches and methods, and current knowledge gaps for the provision of force feedback for RCI systems was performed. Afterward, a real-time tip force estimation method based on image-based shape-sensing and learning-from-simulation was developed and validated. The learning-based model was fairly accurate but required a large database for training which was computationally expensive. Next, a new mechanistic model, i.e., finite arc method (FAM) for soft robots was proposed, formulated, solved, and validated that allowed for fast and accurate modeling of catheter deformation. With FAM, the required training database for the proposed learning-from-simulation method would be generated with high speed and accuracy. In the end, to robustly relay the estimated forces from real-time imaging from the follower robot to the leader haptic device, a novel impedance-based force feedback rendering modality was proposed and implemented on a representative teleoperated RCI system for experimental validation. The proposed method was compared with the classical direct force reflection method and showed enhanced stability, robustness, and accuracy in the presence of communication disruption. The results of this thesis showed that the performance of the proposed integrated force feedback rendering system was in fair compliance with the clinical requirements and had superior robustness compared to the classical direct force reflection method

Restoring Application Traffic of Latency-Sensitive Networked Systems using Adversarial Autoencoders

The Internet of Things (IoT), coupled with the edge computing paradigm, is enabling several pervasive networked applications with stringent real-time requirements, such as telemedicine and haptic telecommunications. Recent advances in network virtualization and artificial intelligence are helping solve network latency and capacity problems, learning from several states of the network stack. However, despite such advances, a network architecture able to meet the demands of next-generation networked applications with stringent real-time requirements still has untackled challenges. In this paper, we argue that only using network (or transport) layer information to predict traffic evolution and other network states may be insufficient, and a more holistic approach that considers predictions of application-layer states is needed to repair the inefficiencies of the TCP/IP architecture. Based on this intuition, we present the design and implementation of Reparo. At its core, the design of our solution is based on the detection of a packet loss and its restoration using a Hidden Markov Model (HMM) empowered with adversarial autoencoders. In our evaluation, we considered a telemedicine use case, specifically a telepathology session, in which a microscope is controlled remotely in real-time to assess histological imagery. Our results confirm that the use of adversarial autoencoders enhances the accuracy of the prediction method satisfying our telemedicine application’s requirements with a notable improvement in terms of throughput and latency perceived by the user

Networking Architecture and Key Technologies for Human Digital Twin in Personalized Healthcare: A Comprehensive Survey

Digital twin (DT), refers to a promising technique to digitally and accurately represent actual physical entities. One typical advantage of DT is that it can be used to not only virtually replicate a system's detailed operations but also analyze the current condition, predict future behaviour, and refine the control optimization. Although DT has been widely implemented in various fields, such as smart manufacturing and transportation, its conventional paradigm is limited to embody non-living entities, e.g., robots and vehicles. When adopted in human-centric systems, a novel concept, called human digital twin (HDT) has thus been proposed. Particularly, HDT allows in silico representation of individual human body with the ability to dynamically reflect molecular status, physiological status, emotional and psychological status, as well as lifestyle evolutions. These prompt the expected application of HDT in personalized healthcare (PH), which can facilitate remote monitoring, diagnosis, prescription, surgery and rehabilitation. However, despite the large potential, HDT faces substantial research challenges in different aspects, and becomes an increasingly popular topic recently. In this survey, with a specific focus on the networking architecture and key technologies for HDT in PH applications, we first discuss the differences between HDT and conventional DTs, followed by the universal framework and essential functions of HDT. We then analyze its design requirements and challenges in PH applications. After that, we provide an overview of the networking architecture of HDT, including data acquisition layer, data communication layer, computation layer, data management layer and data analysis and decision making layer. Besides reviewing the key technologies for implementing such networking architecture in detail, we conclude this survey by presenting future research directions of HDT

Connected healthcare: Improving patient care using digital health technologies

Now more than ever, traditional healthcare models are being overhauled with digital technologies of Healthcare 4.0 being increasingly adopted. Worldwide, digital devices are improving every stage of the patient care pathway. For one, sensors are being used to monitor patient metrics 24/7, permitting swift diagnosis and interventions. At the treatment stage, 3D printers are currently being investigated for the concept of personalised medicine by allowing patients access to on-demand, customisable therapeutics. Robots are also being explored for treatment, by empowering precision surgery or targeted drug delivery. Within medical logistics, drones are being leveraged to deliver critical treatments to remote areas, collect samples, and even provide emergency aid. To enable seamless integration within healthcare, the Internet of Things technology is being exploited to form closed-loop systems that remotely communicate with one another. This review outlines the most promising healthcare technologies and devices, their strengths, drawbacks, and scopes for clinical adoption