Augmented Reality Assistance for Surgical Interventions using Optical See-Through Head-Mounted Displays

Augmented Reality (AR) offers an interactive user experience via enhancing the real world environment with computer-generated visual cues and other perceptual information. It has been applied to different applications, e.g. manufacturing, entertainment and healthcare, through different AR media. An Optical See-Through Head-Mounted Display (OST-HMD) is a specialized hardware for AR, where the computer-generated graphics can be overlaid directly onto the user's normal vision via optical combiners. Using OST-HMD for surgical intervention has many potential perceptual advantages. As a novel concept, many technical and clinical challenges exist for OST-HMD-based AR to be clinically useful, which motivates the work presented in this thesis. From the technical aspects, we first investigate the display calibration of OST-HMD, which is an indispensable procedure to create accurate AR overlay. We propose various methods to reduce the user-related error, improve robustness of the calibration, and remodel the calibration as a 3D-3D registration problem. Secondly, we devise methods and develop hardware prototype to increase the user's visual acuity of both real and virtual content through OST-HMD, to aid them in tasks that require high visual acuity, e.g. dental procedures. Thirdly, we investigate the occlusion caused by the OST-HMD hardware, which limits the user's peripheral vision. We propose to use alternative indicators to remind the user of unattended environment motion. From the clinical perspective, we identified many clinical use cases where OST-HMD-based AR is potentially helpful, developed applications integrated with current clinical systems, and conducted proof-of-concept evaluations. We first present a "virtual monitor'' for image-guided surgery. It can replace real radiology monitors in the operating room with easier user control and more flexibility in positioning. We evaluated the "virtual monitor'' for simulated percutaneous spine procedures. Secondly, we developed ARssist, an application for the bedside assistant in robotic surgery. The assistant can see the robotic instruments and endoscope within the patient body with ARssist. We evaluated the efficiency, safety and ergonomics of the assistant during two typical tasks: instrument insertion and manipulation. The performance for inexperienced users is significantly improved with ARssist, and for experienced users, the system significantly enhanced their confidence level. Lastly, we developed ARAMIS, which utilizes real-time 3D reconstruction and visualization to aid the laparoscopic surgeon. It demonstrates the concept of "X-ray see-through'' surgery. Our preliminary evaluation validated the application via a peg transfer task, and also showed significant improvement in hand-eye coordination. Overall, we have demonstrated that OST-HMD based AR application provides ergonomic improvements, e.g. hand-eye coordination. In challenging situations or for novice users, the improvements in ergonomic factors lead to improvement in task performance. With continuous effort as a community, optical see-through augmented reality technology will be a useful interventional aid in the near future

Recent Advances in Minimally Invasive Surgery

Minimally invasive surgery has become a common term in visceral as well as gynecologic surgery. It has almost evolved into its own surgical speciality over the past 20 years. Today, being firmly established in every subspeciality of visceral surgery, it is now no longer a distinct skillset, but a fixed part of the armamentarium of surgical options available. In every indication, the advantages of a minimally invasive approach include reduced intraoperative blood loss, less postoperative pain, and shorter rehabilitation times, as well as a marked reduction of overall and surgical postoperative morbidity. In the advent of modern oncologic treatment algorithms, these effects not only lower the immediate impact that an operation has on the patient, but also become important key steps in reducing the side-effects of surgery. Thus, they enable surgery to become a module in modern multi-disciplinary cancer treatment, which blends into multimodular treatment options at different times and prolongs and widens the possibilities available to cancer patients. In this quickly changing environment, the requirement to learn and refine not only open surgical but also different minimally invasive techniques on high levels deeply impact modern surgical training pathways. The use of modern elearning tools and new and praxis-based surgical training possibilities have been readily integrated into modern surgical education,which persists throughout the whole surgical career of modern gynecologic and visceral surgery specialists