Intraoperative Planning and Execution of Arbitrary Orthopedic Interventions Using Handheld Robotics and Augmented Reality

The focus of this work is a generic, intraoperative and image-free planning and execution application for arbitrary orthopedic interventions using a novel handheld robotic device and optical see-through glasses (AR). This medical CAD application enables the surgeon to intraoperatively plan the intervention directly on the patient’s bone. The glasses and all the other instruments are accurately calibrated using new techniques. Several interventions show the effectiveness of this approach

Robots and tools for remodeling bone

The field of robotic surgery has progressed from small teams of researchers repurposing industrial robots, to a competitive and highly innovative subsection of the medical device industry. Surgical robots allow surgeons to perform tasks with greater ease, accuracy, or safety, and fall under one of four levels of autonomy; active, semi-active, passive, and remote manipulator. The increased accuracy afforded by surgical robots has allowed for cementless hip arthroplasty, improved postoperative alignment following knee arthroplasty, and reduced duration of intraoperative fluoroscopy among other benefits. Cutting of bone has historically used tools such as hand saws and drills, with other elaborate cutting tools now used routinely to remodel bone. Improvements in cutting accuracy and additional options for safety and monitoring during surgery give robotic surgeries some advantages over conventional techniques. This article aims to provide an overview of current robots and tools with a common target tissue of bone, proposes a new process for defining the level of autonomy for a surgical robot, and examines future directions in robotic surgery

Augmented reality (AR) for surgical robotic and autonomous systems: State of the art, challenges, and solutions

Despite the substantial progress achieved in the development and integration of augmented reality (AR) in surgical robotic and autonomous systems (RAS), the center of focus in most devices remains on improving end-effector dexterity and precision, as well as improved access to minimally invasive surgeries. This paper aims to provide a systematic review of different types of state-of-the-art surgical robotic platforms while identifying areas for technological improvement. We associate specific control features, such as haptic feedback, sensory stimuli, and human-robot collaboration, with AR technology to perform complex surgical interventions for increased user perception of the augmented world. Current researchers in the field have, for long, faced innumerable issues with low accuracy in tool placement around complex trajectories, pose estimation, and difficulty in depth perception during two-dimensional medical imaging. A number of robots described in this review, such as Novarad and SpineAssist, are analyzed in terms of their hardware features, computer vision systems (such as deep learning algorithms), and the clinical relevance of the literature. We attempt to outline the shortcomings in current optimization algorithms for surgical robots (such as YOLO and LTSM) whilst providing mitigating solutions to internal tool-to-organ collision detection and image reconstruction. The accuracy of results in robot end-effector collisions and reduced occlusion remain promising within the scope of our research, validating the propositions made for the surgical clearance of ever-expanding AR technology in the future

Recent Advancements in Augmented Reality for Robotic Applications: A Survey

Robots are expanding from industrial applications to daily life, in areas such as medical robotics, rehabilitative robotics, social robotics, and mobile/aerial robotics systems. In recent years, augmented reality (AR) has been integrated into many robotic applications, including medical, industrial, human–robot interactions, and collaboration scenarios. In this work, AR for both medical and industrial robot applications is reviewed and summarized. For medical robot applications, we investigated the integration of AR in (1) preoperative and surgical task planning; (2) image-guided robotic surgery; (3) surgical training and simulation; and (4) telesurgery. AR for industrial scenarios is reviewed in (1) human–robot interactions and collaborations; (2) path planning and task allocation; (3) training and simulation; and (4) teleoperation control/assistance. In addition, the limitations and challenges are discussed. Overall, this article serves as a valuable resource for working in the field of AR and robotic research, offering insights into the recent state of the art and prospects for improvement

Toward Improving Safety in Neurosurgery with an Active Handheld Instrument

Microsurgical procedures, such as petroclival meningioma resection, require careful surgical actions in order to remove tumor tissue, while avoiding brain and vessel damaging. Such procedures are currently performed under microscope magnification. Robotic tools are emerging in order to filter surgeons’ unintended movements and prevent tools from entering forbidden regions such as vascular structures. The present work investigates the use of a handheld robotic tool (Micron) to automate vessel avoidance in microsurgery. In particular, we focused on vessel segmentation, implementing a deep-learning-based segmentation strategy in microscopy images, and its integration with a feature-based passive 3D reconstruction algorithm to obtain accurate and robust vessel position. We then implemented a virtual-fixture-based strategy to control the handheld robotic tool and perform vessel avoidance. Clay vascular phantoms, lying on a background obtained from microscopy images recorded during petroclival meningioma surgery, were used for testing the segmentation and control algorithms. When testing the segmentation algorithm on 100 different phantom images, a median Dice similarity coefficient equal to 0.96 was achieved. A set of 25 Micron trials of 80 s in duration, each involving the interaction of Micron with a different vascular phantom, were recorded, with a safety distance equal to 2 mm, which was comparable to the median vessel diameter. Micron’s tip entered the forbidden region 24% of the time when the control algorithm was active. However, the median penetration depth was 16.9 μm, which was two orders of magnitude lower than median vessel diameter. Results suggest the system can assist surgeons in performing safe vessel avoidance during neurosurgical procedures

Medical robots with potential applications in participatory and opportunistic remote sensing: A review

Among numerous applications of medical robotics, this paper concentrates on the design, optimal use and maintenance of the related technologies in the context of healthcare, rehabilitation and assistive robotics, and provides a comprehensive review of the latest advancements in the foregoing field of science and technology, while extensively dealing with the possible applications of participatory and opportunistic mobile sensing in the aforementioned domains. The main motivation for the latter choice is the variety of such applications in the settings having partial contributions to functionalities such as artery, radiosurgery, neurosurgery and vascular intervention. From a broad perspective, the aforementioned applications can be realized via various strategies and devices benefiting from detachable drives, intelligent robots, human-centric sensing and computing, miniature and micro-robots. Throughout the paper tens of subjects, including sensor-fusion, kinematic, dynamic and 3D tissue models are discussed based on the existing literature on the state-of-the-art technologies. In addition, from a managerial perspective, topics such as safety monitoring, security, privacy and evolutionary optimization of the operational efficiency are reviewed

Augmented reality for computer assisted orthopaedic surgery

In recent years, computer-assistance and robotics have established their presence in operating theatres and found success in orthopaedic procedures. Benefits of computer assisted orthopaedic surgery (CAOS) have been thoroughly explored in research, finding improvements in clinical outcomes, through increased control and precision over surgical actions. However, human-computer interaction in CAOS remains an evolving field, through emerging display technologies including augmented reality (AR) – a fused view of the real environment with virtual, computer-generated holograms. Interactions between clinicians and patient-specific data generated during CAOS are limited to basic 2D interactions on touchscreen monitors, potentially creating clutter and cognitive challenges in surgery. Work described in this thesis sought to explore the benefits of AR in CAOS through: an integration between commercially available AR and CAOS systems, creating a novel AR-centric surgical workflow to support various tasks of computer-assisted knee arthroplasty, and three pre–clinical studies exploring the impact of the new AR workflow on both existing and newly proposed quantitative and qualitative performance metrics. Early research focused on cloning the (2D) user-interface of an existing CAOS system onto a virtual AR screen and investigating any resulting impacts on usability and performance. An infrared-based registration system is also presented, describing a protocol for calibrating commercial AR headsets with optical trackers, calculating a spatial transformation between surgical and holographic coordinate frames. The main contribution of this thesis is a novel AR workflow designed to support computer-assisted patellofemoral arthroplasty. The reported workflow provided 3D in-situ holographic guidance for CAOS tasks including patient registration, pre-operative planning, and assisted-cutting. Pre-clinical experimental validation on a commercial system (NAVIO®, Smith & Nephew) for these contributions demonstrates encouraging early-stage results showing successful deployment of AR to CAOS systems, and promising indications that AR can enhance the clinician’s interactions in the future. The thesis concludes with a summary of achievements, corresponding limitations and future research opportunities.Open Acces

Development of a Novel Handheld Device for Active Compensation of Physiological Tremor

In microsurgery, the human hand imposes certain limitations in accurately positioning the tip of a device such as scalpel. Any errors in the motion of the hand make microsurgical procedures difficult and involuntary motions such as hand tremors can make some procedures significantly difficult to perform. This is particularly true in the case of vitreoretinal microsurgery. The most familiar source of involuntary motion is physiological tremor. Real-time compensation of tremor is, therefore, necessary to assist surgeons to precisely position and manipulate the tool-tip to accurately perform a microsurgery. In this thesis, a novel handheld device (AID) is described for compensation of physiological tremor in the hand. MEMS-based accelerometers and gyroscopes have been used for sensing the motion of the hand in six degrees of freedom (DOF). An augmented state complementary Kalman filter is used to calculate 2 DOF orientation. An adaptive filtering algorithm, band-limited Multiple Fourier linear combiner (BMFLC), is used to calculate the tremor component in the hand in real-time. Ionic Polymer Metallic Composites (IPMCs) have been used as actuators for deflecting the tool-tip to compensate for the tremor