Recent Developments and Future Challenges in Medical Mixed Reality

As AR technology matures, we have seen many applicationsemerge in entertainment, education and training. However, the useof AR is not yet common in medical practice, despite the great po-tential of this technology to help not only learning and training inmedicine, but also in assisting diagnosis and surgical guidance. Inthis paper, we present recent trends in the use of AR across all med-ical specialties and identify challenges that must be overcome tonarrow the gap between academic research and practical use of ARin medicine. A database of 1403 relevant research papers publishedover the last two decades has been reviewed by using a novel re-search trend analysis method based on text mining algorithm. Wesemantically identified 10 topics including varies of technologiesand applications based on the non-biased and in-personal cluster-ing results from the Latent Dirichlet Allocatio (LDA) model andanalysed the trend of each topic from 1995 to 2015. The statisticresults reveal a taxonomy that can best describes the developmentof the medical AR research during the two decades. And the trendanalysis provide a higher level of view of how the taxonomy haschanged and where the focus will goes. Finally, based on the valu-able results, we provide a insightful discussion to the current limi-tations, challenges and future directions in the field. Our objectiveis to aid researchers to focus on the application areas in medicalAR that are most needed, as well as providing medical practitioners with latest technology advancements

Modular framework for a breast biopsy smart navigation system

Dissertação de mestrado em Informatics EngineeringBreast cancer is currently one of the most commonly diagnosed cancers and the fifth leading cause of cancer-related deaths. Its treatment has a higher survivorship rate when diagnosed in the disease’s early stages. The screening procedure uses medical imaging techniques, such as mammography or ultrasound, to discover possible lesions. When a physician finds a lesion that is likely to be malignant, a biopsy is performed to obtain a sample and determine its characteristics. Currently, real-time ultrasound is the preferred medical imaging modality to perform this procedure. The breast biopsy procedure is highly reliant on the operator’s skill and experience, due to the difficulty in interpreting ultrasound images and correctly aiming the needle. Robotic solutions, and the usage of automatic lesion segmentation in ultrasound imaging along with advanced visualization techniques, such as augmented reality, can potentially make this process simpler, safer, and faster. The OncoNavigator project, in which this dissertation integrates, aims to improve the precision of the current breast cancer interventions. To accomplish this objective various medical training and robotic biopsy aid were developed. An augmented reality ultrasound training solution was created and the device’s tracking capabilities were validated by comparing it with an electromagnetic tracking device. Another solution for ultrasound-guided breast biopsy assisted with augmented reality was developed. This solution displays real-time ultrasound video, automatic lesion segmentation, and biopsy needle trajectory display in the user’s field of view. The validation of this solution was made by comparing its usability with the traditional procedure. A modular software framework was also developed that focuses on the integration of a collaborative medical robot with real-time ultrasound imaging and automatic lesion segmentation. Overall, the developed solutions offered good results. The augmented reality glasses tracking capabilities proved to be as capable as the electromagnetic system, and the augmented reality assisted breast biopsy proved to make the procedure more accurate and precise than the traditional system.O cancro da mama é, atualmente, um dos tipos de cancro mais comuns a serem diagnosticados e a quinta principal causa de mortes relacionadas ao cancro. O seu tratamento tem maior taxa de sobrevivência quando é diagnosticado nas fases iniciais da doença. O procedimento de triagem utiliza técnicas de imagem médica, como mamografia ou ultrassom, para descobrir possíveis lesões. Quando um médico encontra uma lesão com probabilidade de ser maligna, é realizada uma biópsia para obter uma amostra e determinar as suas características. O ultrassom em tempo real é a modalidade de imagem médica preferida para realizar esse procedimento. A biópsia mamária depende da habilidade e experiência do operador, devido à dificuldade de interpretação das imagens ultrassonográficas e ao direcionamento correto da agulha. Soluções robóticas, com o uso de segmentação automática de lesões em imagens de ultrassom, juntamente com técnicas avançadas de visualização, nomeadamente realidade aumentada, podem tornar esse processo mais simples, seguro e rápido. O projeto OncoNavigator, que esta dissertação integra, visa melhorar a precisão das atuais intervenções ao cancro da mama. Para atingir este objetivo, vários ajudas para treino médico e auxílio à biópsia por meio robótico foram desenvolvidas. Uma solução de treino de ultrassom com realidade aumentada foi criada e os recursos de rastreio do dispositivo foram validados comparando-os com um dispositivo eletromagnético. Outra solução para biópsia de mama guiada por ultrassom assistida com realidade aumentada foi desenvolvida. Esta solução exibe vídeo de ultrassom em tempo real, segmentação automática de lesões e exibição da trajetória da agulha de biópsia no campo de visão do utilizador. A validação desta solução foi feita comparando a sua usabilidade com o procedimento tradicional. Também foi desenvolvida uma estrutura de software modular que se concentra na integração de um robô médico colaborativo com imagens de ultrassom em tempo real e segmentação automática de lesões. Os recursos de rastreio dos óculos de realidade aumentada mostraram-se tão capazes quanto o sistema eletromagnético, e a biópsia de mama assistida por realidade aumentada provou tornar o procedimento mais exato e preciso do que o sistema tradicional

Systems and technologies for objective evaluation of technical skills in laparoscopic surgery

Minimally invasive surgery is a highly demanding surgical approach regarding technical requirements for the surgeon, who must be trained in order to perform a safe surgical intervention. Traditional surgical education in minimally invasive surgery is commonly based on subjective criteria to quantify and evaluate surgical abilities, which could be potentially unsafe for the patient. Authors, surgeons and associations are increasingly demanding the development of more objective assessment tools that can accredit surgeons as technically competent. This paper describes the state of the art in objective assessment methods of surgical skills. It gives an overview on assessment systems based on structured checklists and rating scales, surgical simulators, and instrument motion analysis. As a future work, an objective and automatic assessment method of surgical skills should be standardized as a means towards proficiency-based curricula for training in laparoscopic surgery and its certification

From Concept to Market: Surgical Robot Development

Surgical robotics and supporting technologies have really become a prime example of modern applied information technology infiltrating our everyday lives. The development of these systems spans across four decades, and only the last few years brought the market value and saw the rising customer base imagined already by the early developers. This chapter guides through the historical development of the most important systems, and provide references and lessons learnt for current engineers facing similar challenges. A special emphasis is put on system validation, assessment and clearance, as the most commonly cited barrier hindering the wider deployment of a system

Haptics-Enabled Teleoperation for Robotics-Assisted Minimally Invasive Surgery

The lack of force feedback (haptics) in robotic surgery can be considered to be a safety risk leading to accidental tissue damage and puncturing of blood vessels due to excessive forces being applied to tissue and vessels or causing inefficient control over the instruments because of insufficient applied force. This project focuses on providing a satisfactory solution for introducing haptic feedback in robotics-assisted minimally invasive surgical (RAMIS) systems. The research addresses several key issues associated with the incorporation of haptics in a master-slave (teleoperated) robotic environment for minimally invasive surgery (MIS). In this project, we designed a haptics-enabled dual-arm (two masters - two slaves) robotic MIS testbed to investigate and validate various single-arm as well as dual-arm teleoperation scenarios. The most important feature of this setup is the capability of providing haptic feedback in all 7 degrees of freedom (DOF) required for RAMIS (3 translations, 3 rotations and pinch motion of the laparoscopic tool). The setup also enables the evaluation of the effect of replacing haptic feedback by other sensory cues such as visual representation of haptic information (sensory substitution) and the hypothesis that surgical outcomes may be improved by substituting or augmenting haptic feedback by such sensory cues

Software Design of an Experimental Management Suite for Evaluating Time-Delayed Teleoperative Simulations

A means of assessing human performance as it relates to telesurgery is critical in an age where computer-assisted surgery is becoming more commonplace in operating rooms around the world. This is particularly true when a human is controlling the robotic instruments over a considerable geographic distance; unpredictable delays in data transmission over a network can degrade the human-computer system performance. As the delay is increased, so too will the overall time to complete a given task along with its associated error rate. However, objective measures on the effect on performance are needed. The methodology developed here is based on Fitts’ paradigm; a framework that can be used to quantify human performance under simulated latency conditions. Data gathered from the software developed in this thesis shows a strong, positive, linear correlation between a subject’s performance and the imposed task latency. This coincides with similar studies performed using the same paradigm, demonstrating the usefulness of such a methodology with respect to systems for telesurgery and training

Augmented Reality Assistance for Surgical Interventions using Optical See-Through Head-Mounted Displays

Augmented Reality (AR) offers an interactive user experience via enhancing the real world environment with computer-generated visual cues and other perceptual information. It has been applied to different applications, e.g. manufacturing, entertainment and healthcare, through different AR media. An Optical See-Through Head-Mounted Display (OST-HMD) is a specialized hardware for AR, where the computer-generated graphics can be overlaid directly onto the user's normal vision via optical combiners. Using OST-HMD for surgical intervention has many potential perceptual advantages. As a novel concept, many technical and clinical challenges exist for OST-HMD-based AR to be clinically useful, which motivates the work presented in this thesis. From the technical aspects, we first investigate the display calibration of OST-HMD, which is an indispensable procedure to create accurate AR overlay. We propose various methods to reduce the user-related error, improve robustness of the calibration, and remodel the calibration as a 3D-3D registration problem. Secondly, we devise methods and develop hardware prototype to increase the user's visual acuity of both real and virtual content through OST-HMD, to aid them in tasks that require high visual acuity, e.g. dental procedures. Thirdly, we investigate the occlusion caused by the OST-HMD hardware, which limits the user's peripheral vision. We propose to use alternative indicators to remind the user of unattended environment motion. From the clinical perspective, we identified many clinical use cases where OST-HMD-based AR is potentially helpful, developed applications integrated with current clinical systems, and conducted proof-of-concept evaluations. We first present a "virtual monitor'' for image-guided surgery. It can replace real radiology monitors in the operating room with easier user control and more flexibility in positioning. We evaluated the "virtual monitor'' for simulated percutaneous spine procedures. Secondly, we developed ARssist, an application for the bedside assistant in robotic surgery. The assistant can see the robotic instruments and endoscope within the patient body with ARssist. We evaluated the efficiency, safety and ergonomics of the assistant during two typical tasks: instrument insertion and manipulation. The performance for inexperienced users is significantly improved with ARssist, and for experienced users, the system significantly enhanced their confidence level. Lastly, we developed ARAMIS, which utilizes real-time 3D reconstruction and visualization to aid the laparoscopic surgeon. It demonstrates the concept of "X-ray see-through'' surgery. Our preliminary evaluation validated the application via a peg transfer task, and also showed significant improvement in hand-eye coordination. Overall, we have demonstrated that OST-HMD based AR application provides ergonomic improvements, e.g. hand-eye coordination. In challenging situations or for novice users, the improvements in ergonomic factors lead to improvement in task performance. With continuous effort as a community, optical see-through augmented reality technology will be a useful interventional aid in the near future

ROBOTIC TELESURGERY: AN INVESTIGATION OF UTILITY, HUMAN ADAPTATION, AND PERFORMANCE

Robotic surgery is a powerful, new method for performing minimally invasive surgery (MIS). The method allows complex procedures through incisions which are 10 mm or less. Robotic surgery has grown rapidly because small MIS incisions result in rapid patient recovery compared to conventional methods. Although surgical robots have the potential of long distance control, insufficient data is available to determine whether long distance robotic surgery, or telesurgery, is practical. Telesurgery could provide multiple benefits, including dissemination of expertise, widespread patient care, cost savings, and improved community care. We describe a series of experiments to investigate telesurgery using a one of a kind telesurgery platform and ground- and satellite-based Internet networks. The networks provided the redundancy and quality of service that would be required for human surgery. Tolerances for performing surgical tasks over a long distance were unknown. We show that operators using the platform can complete dry lab manoeuvres with communication latencies up to 500 ms, with no appreciable increase in error rates. Such latency would be equivalent to a North American transcontinental distance, implying a wide range of telesurgical capability. The characteristics of ground- and satellite-based Internet networks for telesurgery were unavailable. We demonstrate that emulated surgery in animals can be effectively performed using either ground or satellite. The networks can reliably support surgery, and satellite-based surgery can be performed even though latency exceeds 500 ms. Further, satellite bandwidth should be above 5 Mb/s for telesurgery applications. Satellite networks could be used either for back up or primarily where a community does not have ground-based equipment. iii Methods of training operators for telesurgery had not been explored. We demonstrate two methods of training for telesurgery. Operators doing dry lab surgical manoeuvres performed equally well either with sequentially increasing latency or with full latency only, suggesting that both methods of training may be effective. Telesurgery can become a practical method of treatment. Within a few years, more widespread platforms and telecommunications may exist to launch everyday telesurgery procedures

5G Smart and innovative Healthcare services: opportunities, challenges and prospective solutions

Due to its abilities to boost productivity, reduce costs and enhance user experiences, smart healthcare is widely recognised as a potential solution to reduce pressures on existing health systems. Since the new era of 5G will unite enhanced connectivity, improved cloud-based storage and interconnection of an array of devices and services, a massive boost in the digital transformation of healthcare is expected. In this transformation process, healthcare services such as medical diagnosis, treatment and remote surgery will be facilitated by a range of technologies such as Internet of Things, Robotics and Artificial Intelligence, among others, that will advance further under 5G. Moreover, real-time health services will become a reality and will offer people with quality care and improved experiences. On the other hand, different challenges can hinder the proliferation of 5G smart and innovative healthcare solutions, including security and heterogeneous devices. This chapter presents how 5G will boost digital transformation of healthcare through delivery and consumption of smart and innovative healthcare services, while probing into key hurdles in the process as well as prospective solutions