Interfaces for Modular Surgical Planning and Assistance Systems

Modern surgery of the 21st century relies in many aspects on computers or, in a wider sense, digital data processing. Department administration, OR scheduling, billing, and - with increasing pervasion - patient data management are performed with the aid of so called Surgical Information Systems (SIS) or, more general, Hospital Information Systems (HIS). Computer Assisted Surgery (CAS) summarizes techniques which assist a surgeon in the preparation and conduction of surgical interventions. Today still predominantly based on radiology images, these techniques include the preoperative determination of an optimal surgical strategy and intraoperative systems which aim at increasing the accuracy of surgical manipulations. CAS is a relatively young field of computer science. One of the unsolved "teething troubles" of CAS is the absence of technical standards for the interconnectivity of CAS system. Current CAS systems are usually "islands of information" with no connection to other devices within the operating room or hospital-wide information systems. Several workshop reports and individual publications point out that this situation leads to ergonomic, logistic, and economic limitations in hospital work. Perioperative processes are prolonged by the manual installation and configuration of an increasing amount of technical devices. Intraoperatively, a large amount of the surgeons'' attention is absorbed by the requirement to monitor and operate systems. The need for open infrastructures which enable the integration of CAS devices from different vendors in order to exchange information as well as commands among these devices through a network has been identified by numerous experts with backgrounds in medicine as well as engineering. This thesis contains two approaches to the integration of CAS systems: - For perioperative data exchange, the specification of new data structures as an amendment to the existing DICOM standard for radiology image management is presented. The extension of DICOM towards surgical application allows for the seamless integration of surgical planning and reporting systems into DICOM-based Picture Archiving and Communication Systems (PACS) as they are installed in most hospitals for the exchange and long-term archival of patient images and image-related patient data. - For the integration of intraoperatively used CAS devices, such as, e.g., navigation systems, video image sources, or biosensors, the concept of a surgical middleware is presented. A c++ class library, the TiCoLi, is presented which facilitates the configuration of ad-hoc networks among the modules of a distributed CAS system as well as the exchange of data streams, singular data objects, and commands between these modules. The TiCoLi is the first software library for a surgical field of application to implement all of these services. To demonstrate the suitability of the presented specifications and their implementation, two modular CAS applications are presented which utilize the proposed DICOM extensions for perioperative exchange of surgical planning data as well as the TiCoLi for establishing an intraoperative network of autonomous, yet not independent, CAS modules.Die moderne Hochleistungschirurgie des 21. Jahrhunderts ist auf vielerlei Weise abhängig von Computern oder, im weiteren Sinne, der digitalen Datenverarbeitung. Administrative Abläufe, wie die Erstellung von Nutzungsplänen für die verfügbaren technischen, räumlichen und personellen Ressourcen, die Rechnungsstellung und - in zunehmendem Maße - die Verwaltung und Archivierung von Patientendaten werden mit Hilfe von digitalen Informationssystemen rationell und effizient durchgeführt. Innerhalb der Krankenhausinformationssysteme (KIS, oder englisch HIS) stehen für die speziellen Bedürfnisse der einzelnen Fachabteilungen oft spezifische Informationssysteme zur Verfügung. Chirurgieinformationssysteme (CIS, oder englisch SIS) decken hierbei vor allen Dingen die Bereiche Operationsplanung sowie Materialwirtschaft für spezifisch chirurgische Verbrauchsmaterialien ab. Während die genannten HIS und SIS vornehmlich der Optimierung administrativer Aufgaben dienen, stehen die Systeme der Computerassistierten Chirugie (CAS) wesentlich direkter im Dienste der eigentlichen chirugischen Behandlungsplanung und Therapie. Die CAS verwendet Methoden der Robotik, digitalen Bild- und Signalverarbeitung, künstlichen Intelligenz, numerischen Simulation, um nur einige zu nennen, zur patientenspezifischen Behandlungsplanung und zur intraoperativen Unterstützung des OP-Teams, allen voran des Chirurgen. Vor allen Dingen Fortschritte in der räumlichen Verfolgung von Werkzeugen und Patienten ("Tracking"), die Verfügbarkeit dreidimensionaler radiologischer Aufnahmen (CT, MRT, ...) und der Einsatz verschiedener Robotersysteme haben in den vergangenen Jahrzehnten den Einzug des Computers in den Operationssaal - medienwirksam - ermöglicht. Weniger prominent, jedoch keinesfalls von untergeordnetem praktischen Nutzen, sind Beispiele zur automatisierten Überwachung klinischer Messwerte, wie etwa Blutdruck oder Sauerstoffsättigung. Im Gegensatz zu den meist hochgradig verteilten und gut miteinander verwobenen Informationssystemen für die Krankenhausadministration und Patientendatenverwaltung, sind die Systeme der CAS heutzutage meist wenig oder überhaupt nicht miteinander und mit Hintergrundsdatenspeichern vernetzt. Eine Reihe wissenschaftlicher Publikationen und interdisziplinärer Workshops hat sich in den vergangen ein bis zwei Jahrzehnten mit den Problemen des Alltagseinsatzes von CAS Systemen befasst. Mit steigender Intensität wurde hierbei auf den Mangel an infrastrukturiellen Grundlagen für die Vernetzung intraoperativ eingesetzter CAS Systeme miteinander und mit den perioperativ eingesetzten Planungs-, Dokumentations- und Archivierungssystemen hingewiesen. Die sich daraus ergebenden negativen Einflüsse auf die Effizienz perioperativer Abläufe - jedes Gerät muss manuell in Betrieb genommen und mit den spezifischen Daten des nächsten Patienten gefüttert werden - sowie die zunehmende Aufmerksamkeit, welche der Operateur und sein Team auf die Überwachung und dem Betrieb der einzelnen Geräte verwenden muss, werden als eine der "Kinderkrankheiten" dieser relativ jungen Technologie betrachtet und stehen einer Verbreitung über die Grenzen einer engagierten technophilen Nutzergruppe hinaus im Wege. Die vorliegende Arbeit zeigt zwei parallel von einander (jedoch, im Sinne der Schnittstellenkompatibilität, nicht gänzlich unabhängig voneinander) zu betreibende Ansätze zur Integration von CAS Systemen. - Für den perioperativen Datenaustausch wird die Spezifikation zusätzlicher Datenstrukturen zum Transfer chirurgischer Planungsdaten im Rahmen des in radiologischen Bildverarbeitungssystemen weit verbreiteten DICOM Standards vorgeschlagen und an zwei Beispielen vorgeführt. Die Erweiterung des DICOM Standards für den perioperativen Einsatz ermöglicht hierbei die nahtlose Integration chirurgischer Planungssysteme in existierende "Picture Archiving and Communication Systems" (PACS), welche in den meisten Fällen auf dem DICOM Standard basieren oder zumindest damit kompatibel sind. Dadurch ist einerseits der Tatsache Rechnung getragen, dass die patientenspezifische OP-Planung in hohem Masse auf radiologischen Bildern basiert und andererseits sicher gestellt, dass die Planungsergebnisse entsprechend der geltenden Bestimmungen langfristig archiviert und gegen unbefugten Zugriff geschützt sind - PACS Server liefern hier bereits wohlerprobte Lösungen. - Für die integration intraoperativer CAS Systeme, wie etwa Navigationssysteme, Videobildquellen oder Sensoren zur Überwachung der Vitalparameter, wird das Konzept einer "chirurgischen Middleware" vorgestellt. Unter dem Namen TiCoLi wurde eine c++ Klassenbibliothek entwickelt, auf deren Grundlage die Konfiguration von ad-hoc Netzwerken während der OP-Vorbereitung mittels plug-and-play Mechanismen erleichtert wird. Nach erfolgter Konfiguration ermöglicht die TiCoLi den Austausch kontinuierlicher Datenströme sowie einzelner Datenpakete und Kommandos zwischen den Modulen einer verteilten CAS Anwendung durch ein Ethernet-basiertes Netzwerk. Die TiCoLi ist die erste frei verfügbare Klassenbibliothek welche diese Funktionalitäten dediziert für einen Einsatz im chirurgischen Umfeld vereinigt. Zum Nachweis der Tauglichkeit der gezeigten Spezifikationen und deren Implementierungen, werden zwei modulare CAS Anwendungen präsentiert, welche die vorgeschlagenen DICOM Erweiterungen zum perioperativen Austausch von Planungsergebnissen sowie die TiCoLi zum intraoperativen Datenaustausch von Messdaten unter echzeitnahen Anforderungen verwenden

Mobile and Low-cost Hardware Integration in Neurosurgical Image-Guidance

It is estimated that 13.8 million patients per year require neurosurgical interventions worldwide, be it for a cerebrovascular disease, stroke, tumour resection, or epilepsy treatment, among others. These procedures involve navigating through and around complex anatomy in an organ where damage to eloquent healthy tissue must be minimized. Neurosurgery thus has very specific constraints compared to most other domains of surgical care. These constraints have made neurosurgery particularly suitable for integrating new technologies. Any new method that has the potential to improve surgical outcomes is worth pursuing, as it has the potential to not only save and prolong lives of patients, but also increase the quality of life post-treatment. In this thesis, novel neurosurgical image-guidance methods are developed, making use of currently available, low-cost off-the-shelf components. In particular, a mobile device (e.g. smartphone or tablet) is integrated into a neuronavigation framework to explore new augmented reality visualization paradigms and novel intuitive interaction methods. The developed tools aim at improving image-guidance using augmented reality to improve intuitiveness and ease of use. Further, we use gestures on the mobile device to increase interactivity with the neuronavigation system in order to provide solutions to the problem of accuracy loss or brain shift that occurs during surgery. Lastly, we explore the effectiveness and accuracy of low-cost hardware components (i.e. tracking systems and ultrasound) that could be used to replace the current high cost hardware that are integrated into commercial image-guided neurosurgery systems. The results of our work show the feasibility of using mobile devices to improve neurosurgical processes. Augmented reality enables surgeons to focus on the surgical field while getting intuitive guidance information. Mobile devices also allow for easy interaction with the neuronavigation system thus enabling surgeons to directly interact with systems in the operating room to improve accuracy and streamline procedures. Lastly, our results show that low-cost components can be integrated into a neurosurgical guidance system at a fraction of the cost, while having a negligible impact on accuracy. The developed methods have the potential to improve surgical workflows, as well as democratize access to higher quality care worldwide

Personalized medicine in surgical treatment combining tracking systems, augmented reality and 3D printing

Mención Internacional en el título de doctorIn the last twenty years, a new way of practicing medicine has been focusing on the problems and needs of each patient as an individual thanks to the significant advances in healthcare technology, the so-called personalized medicine. In surgical treatments, personalization has been possible thanks to key technologies adapted to the specific anatomy of each patient and the needs of the physicians. Tracking systems, augmented reality (AR), three-dimensional (3D) printing and artificial intelligence (AI) have previously supported this individualized medicine in many ways. However, their independent contributions show several limitations in terms of patient-to-image registration, lack of flexibility to adapt to the requirements of each case, large preoperative planning times, and navigation complexity. The main objective of this thesis is to increase patient personalization in surgical treatments by combining these technologies to bring surgical navigation to new complex cases by developing new patient registration methods, designing patient-specific tools, facilitating access to augmented reality by the medical community, and automating surgical workflows. In the first part of this dissertation, we present a novel framework for acral tumor resection combining intraoperative open-source navigation software, based on an optical tracking system, and desktop 3D printing. We used additive manufacturing to create a patient-specific mold that maintained the same position of the distal extremity during image-guided surgery as in the preoperative images. The feasibility of the proposed workflow was evaluated in two clinical cases (soft-tissue sarcomas in hand and foot). We achieved an overall accuracy of the system of 1.88 mm evaluated on the patient-specific 3D printed phantoms. Surgical navigation was feasible during both surgeries, allowing surgeons to verify the tumor resection margin. Then, we propose and augmented reality navigation system that uses 3D printed surgical guides with a tracking pattern enabling automatic patient-to-image registration in orthopedic oncology. This specific tool fits on the patient only in a pre-designed location, in this case bone tissue. This solution has been developed as a software application running on Microsoft HoloLens. The workflow was validated on a 3D printed phantom replicating the anatomy of a patient presenting an extraosseous Ewing’s sarcoma, and then tested during the actual surgical intervention. The results showed that the surgical guide with the reference marker can be placed precisely with an accuracy of 2 mm and a visualization error lower than 3 mm. The application allowed physicians to visualize the skin, bone, tumor and medical images overlaid on the phantom and patient. To enable the use of AR and 3D printing by inexperienced users without broad technical knowledge, we designed a step-by-step methodology. The proposed protocol describes how to develop an AR smartphone application that allows superimposing any patient-based 3D model onto a real-world environment using a 3D printed marker tracked by the smartphone camera. Our solution brings AR solutions closer to the final clinical user, combining free and open-source software with an open-access protocol. The proposed guide is already helping to accelerate the adoption of these technologies by medical professionals and researchers. In the next section of the thesis, we wanted to show the benefits of combining these technologies during different stages of the surgical workflow in orthopedic oncology. We designed a novel AR-based smartphone application that can display the patient’s anatomy and the tumor’s location. A 3D printed reference marker, designed to fit in a unique position of the affected bone tissue, enables automatic registration. The system has been evaluated in terms of visualization accuracy and usability during the whole surgical workflow on six realistic phantoms achieving a visualization error below 3 mm. The AR system was tested in two clinical cases during surgical planning, patient communication, and surgical intervention. These results and the positive feedback obtained from surgeons and patients suggest that the combination of AR and 3D printing can improve efficacy, accuracy, and patients’ experience In the final section, two surgical navigation systems have been developed and evaluated to guide electrode placement in sacral neurostimulation procedures based on optical tracking and augmented reality. Our results show that both systems could minimize patient discomfort and improve surgical outcomes by reducing needle insertion time and number of punctures. Additionally, we proposed a feasible clinical workflow for guiding SNS interventions with both navigation methodologies, including automatically creating sacral virtual 3D models for trajectory definition using artificial intelligence and intraoperative patient-to-image registration. To conclude, in this thesis we have demonstrated that the combination of technologies such as tracking systems, augmented reality, 3D printing, and artificial intelligence overcomes many current limitations in surgical treatments. Our results encourage the medical community to combine these technologies to improve surgical workflows and outcomes in more clinical scenarios.Programa de Doctorado en Ciencia y Tecnología Biomédica por la Universidad Carlos III de MadridPresidenta: María Jesús Ledesma Carbayo.- Secretaria: María Arrate Muñoz Barrutia.- Vocal: Csaba Pinte

Intraoperative Planning and Execution of Arbitrary Orthopedic Interventions Using Handheld Robotics and Augmented Reality

The focus of this work is a generic, intraoperative and image-free planning and execution application for arbitrary orthopedic interventions using a novel handheld robotic device and optical see-through glasses (AR). This medical CAD application enables the surgeon to intraoperatively plan the intervention directly on the patient’s bone. The glasses and all the other instruments are accurately calibrated using new techniques. Several interventions show the effectiveness of this approach

Lumbar-sacral pedicle screw insertion with preoperative CT-based navigation

Objectif: Nous avons effectué une étude chez 135 patients ayant subis une chirurgie lombo-sacrée avec vissage pédiculaire sous navigation par tomographie axiale. Nous avons évalué la précision des vis pédiculaires et les résultats cliniques. Méthodes: Cette étude comporte 44 hommes et 91 femmes (âge moyen=61, intervalle 24-90 ans). Les diamètres, longueurs et trajectoires des 836 vis ont été planifiés en préopératoire avec un système de navigation (SNN, Surgical Navigation Network, Mississauga). Les patients ont subi une fusion lombaire (55), lombo-sacrée (73) et thoraco-lombo-sacrée (7). La perforation pédiculaire, la longueur des vis et les spondylolisthesis sont évalués par tomographies axiales postopératoires. Le niveau de douleur est mesuré par autoévaluations, échelles visuelles analogues et questionnaires (Oswestry et SF-36). La fusion osseuse a été évaluée par l’examen des radiographies postopératoires. Résultats: Une perforation des pédicules est présente pour 49/836 (5.9%) des vis (2.4% latéral, 1.7% inférieur, 1.1% supérieur, 0.7% médial). Les erreurs ont été mineures (0.1-2mm, 46/49) ou intermédiaires (2.1 - 4mm, 3/49 en latéral). Il y a aucune erreur majeure (≥ 4.1mm). Certaines vis ont été jugées trop longues (66/836, 8%). Le temps moyen pour insérer une vis en navigation a été de 19.1 minutes de l΄application au retrait du cadre de référence. Un an postopératoire on note une amélioration de la douleur des jambes et lombaire de 72% et 48% en moyenne respectivement. L’amélioration reste stable après 2 ans. La dégénérescence radiologique au dessus et sous la fusion a été retrouvée chez 44 patients (33%) and 3 patients respectivement (2%). Elle est survenue en moyenne 22.2 ± 2.6 mois après la chirurgie. Les fusions se terminant à L2 ont été associées à plus de dégénération (14/25, 56%). Conclusion: La navigation spinale basée sur des images tomographiques préopératoires est une technique sécuritaire et précise. Elle donne de bons résultats à court terme justifiant l’investissement de temps chirurgical. La dégénérescence segmentaire peut avoir un impact négatif sur les résultats radiologique et cliniques.Objective: The authors studied 135 consecutive patients following a lumbo-sacral fixation using pedicle screws and CT-based navigation to evaluate pedicle screw accuracy and clinical outcomes. Methods: The series included 44 men and 91 women (mean age 61 years, range 24-90 years). All 836 screws were planned with pre-operative CT-Scans in a navigation system (SNN, Surgical Navigation Network, Mississauga, Ontario, Canada) for diameter, length and direction. Fixation included the lumbar spines only (55), the lumbo-sacral spine (73) or the thoraco-lumbo-sacral spine (7). Pedicle perforation, screw length and spondylolisthesis were assessed on post-operative CT-Scan. Pain was surveyed using self-rated scales, visual analogue scales, Oswestry and SF-36 questionnaires. Bony union was assessed on post-operative follow-up radiographs. Results: Pedicle perforation was found in 49/836 (5.9%) screws (2.4% laterally, 1.7% inferiorly, 1.1% superiorly, 0.7% medially). The errors were minor (0.1-2mm, 46/49) or intermediate (2.1 – 4 mm, 3/49). All intermediate errors were lateral. There were no major errors (≥ 4.1mm). Some screws were judged too long (66/836, 8%). The average time to insert one screw with navigation was 19.1 minutes from application to removal of the reference frame. The amount of improvement at one year post-operation for self-rated leg and back pain were 72% and 48% respectively. The improvement was stable over 2 years. Above-level and below-level radiological degenerations were found in 44 patients (33%) and 3 patients respectively (2%) and occurred on average 22.2 ± 2.6 months after the surgery. Fusions ending at L2 had the most degenerations (14/25, 56%). Conclusion: CT-based preoperative navigation for lumbo-sacral pedicle screw insertion is accurate and associated with a good short term outcome, making it worth the investment of the additional time required. Segmental degeneration may have a negative effect on radiological and clinical outcomes

Simulation Guided Navigation in cranio-maxillo-facial surgery: a new approach to improve intraoperative three-dimensional accuracy and reproducibility during surgery.

The aim of this PhD thesis " Simulation Guided Navigation in cranio- maxillo- facial surgery : a new approach to Improve intraoperative three-dimensional accuracy and reproducibility during surgery ." was at the center of its attention the various applications of a method introduced by our School in 2010 and has as its theme the increase of interest of reproducibility of surgical programs through methods that in whole or in part are using intraoperative navigation. It was introduced in Orthognathic Surgery Validation a new method for the interventions carried out according to the method Simulation Guided Navigation in facial deformities ; was then analyzed the method of three-dimensional control of the osteotomies through the use of templates and cutting of plates using the method precontoured CAD -CAM and laser sintering . It was finally proceeded to introduce the method of piezonavigated surgery in the various branches of maxillofacial surgery . These studies have been subjected to validation processes and the results are presented .Obiettivo di questa tesi di Dottorato “Simulation Guided Navigation in cranio-maxillo-facial surgery: a new approach to improve intraoperative three-dimensional accuracy and reproducibility during surgery.” ha avuto al centro delle proprie attenzioni le varie applicazioni di una metodica introdotta dalla ns. Scuola nel 2010 e che ha come tema di interesse l’aumento delle riproducibilità dei programmi chirurgici mediante metodiche che in toto o in parte utilizzano il navigatore intraoperatorio. Si è introdotto in Chirurgia Ortognatica un nuovo Metodo di Validazione per gli interventi effettuati secondo la metodica Simulation Guided Navigation nelle malformazioni facciali ; si è poi analizzata la metodica di controllo tridimensionale delle osteotomie mediante l’utilizzo delle dime di taglio e delle placche premodellate mediante metodica CAD-CAM e sinterizzazione laser. Si è infine proceduto ad introdurre la metodica di chirurgia piezonavigata alle varie branche di chirurgia maxillo-facciale. Tali studi sono stati sottoposti a processi di validazione ed i risultati vengono presentati