Augmented Reality and Artificial Intelligence in Image-Guided and Robot-Assisted Interventions

In minimally invasive orthopedic procedures, the surgeon places wires, screws, and surgical implants through the muscles and bony structures under image guidance. These interventions require alignment of the pre- and intra-operative patient data, the intra-operative scanner, surgical instruments, and the patient. Suboptimal interaction with patient data and challenges in mastering 3D anatomy based on ill-posed 2D interventional images are essential concerns in image-guided therapies. State of the art approaches often support the surgeon by using external navigation systems or ill-conditioned image-based registration methods that both have certain drawbacks. Augmented reality (AR) has been introduced in the operating rooms in the last decade; however, in image-guided interventions, it has often only been considered as a visualization device improving traditional workflows. Consequently, the technology is gaining minimum maturity that it requires to redefine new procedures, user interfaces, and interactions. This dissertation investigates the applications of AR, artificial intelligence, and robotics in interventional medicine. Our solutions were applied in a broad spectrum of problems for various tasks, namely improving imaging and acquisition, image computing and analytics for registration and image understanding, and enhancing the interventional visualization. The benefits of these approaches were also discovered in robot-assisted interventions. We revealed how exemplary workflows are redefined via AR by taking full advantage of head-mounted displays when entirely co-registered with the imaging systems and the environment at all times. The proposed AR landscape is enabled by co-localizing the users and the imaging devices via the operating room environment and exploiting all involved frustums to move spatial information between different bodies. The system's awareness of the geometric and physical characteristics of X-ray imaging allows the exploration of different human-machine interfaces. We also leveraged the principles governing image formation and combined it with deep learning and RGBD sensing to fuse images and reconstruct interventional data. We hope that our holistic approaches towards improving the interface of surgery and enhancing the usability of interventional imaging, not only augments the surgeon's capabilities but also augments the surgical team's experience in carrying out an effective intervention with reduced complications

Manufacturing Metrology

Metrology is the science of measurement, which can be divided into three overlapping activities: (1) the definition of units of measurement, (2) the realization of units of measurement, and (3) the traceability of measurement units. Manufacturing metrology originally implicates the measurement of components and inputs for a manufacturing process to assure they are within specification requirements. It can also be extended to indicate the performance measurement of manufacturing equipment. This Special Issue covers papers revealing novel measurement methodologies and instrumentations for manufacturing metrology from the conventional industry to the frontier of the advanced hi-tech industry. Twenty-five papers are included in this Special Issue. These published papers can be categorized into four main groups, as follows: Length measurement: covering new designs, from micro/nanogap measurement with laser triangulation sensors and laser interferometers to very-long-distance, newly developed mode-locked femtosecond lasers. Surface profile and form measurements: covering technologies with new confocal sensors and imagine sensors: in situ and on-machine measurements. Angle measurements: these include a new 2D precision level design, a review of angle measurement with mode-locked femtosecond lasers, and multi-axis machine tool squareness measurement. Other laboratory systems: these include a water cooling temperature control system and a computer-aided inspection framework for CMM performance evaluation

Optical accuracy assessment of robotically assisted dental implant surgery

BACKGROUND: Static and dynamic dental implant guidance systems have established themselves as effective choices that result in predictable and relatively accurate dental implant placement. Generally, studies assess this accuracy using a postoperative CBCT, which has disadvantages such as additional radiation exposure for the patient. This pilot study proposed a scanbody-agnostic method of implant position assessment using intraoral scanning technology and applied it as an accuracy test of robotically assisted dental implant placement using the Neocis Yomi. MATERIALS AND METHODS: All of the robotically assisted dental implant surgery was performed in the Postdoctoral Periodontology clinic at Boston University Henry M. Goldman School of Dental Medicine. Completely edentulous patients were excluded. A total of eleven (11) implants were included in the study, eight (8) of which were fully guided. An optical impression of each implant position was obtained using a CEREC Omnicam (SW 5.1) intraoral scanner. Each sample used either a DESS Lab Scan Body or an Elos Accurate Scan Body as a means to indirectly index the position of the implant. A comparison of planned implant position versus executed surgical implant position was performed for each placement using Geomagic Control X software. Global positional and angular deviations were quantified using a proposed scanbody-agnostic method. Intraoral directionality of deviation was visually qualified by the author (D.K). RESULTS: Mean global positional deviations at the midpoints of the top of each scanbody were 1.7417 mm in the partially guided samples and 1.1300 mm in the fully guided samples. Mean global positional deviations at the midpoints of the restorative platforms of each implant were 1.3142 mm in the partially guided sample and 1.27045 mm in the fully guided samples. Mean global positional deviations at the midpoints of the apex of each implant were 1.455 mm in the partially guided samples and 1.574 mm in the fully guided samples. Mean angular deviations were 3.7492 degrees in the partially guided samples and 2.6432 degrees in the fully guided samples. CONCLUSION: Within the sample size limitations, robotically assisted dental implant surgery offers similar implant placement accuracy compared to published static and dynamic implant placement guidance systems. Intraoral optical assessment of dental implant position used in this study allows comparable analysis to other methods without requiring additional exposure to radiation and should be considered the default method of assessing guidance accuracy

Computer assisted orthopaedic surgery : past, present and future

Computer technology is ubiquitous and relied upon in virtually all professional activities including neurosurgery, which is why it is surprising that it is not the case for orthopaedic surgery with fewer than 5% of surgeons using available computer technology in their procedures. In this review, we explore the evolution and background of Computer Assisted Orthopaedic Surgery (CAOS), delving into the basic principles behind the technology and the changes in the discussion on the subject throughout the years and the impact these discussions had on the field. We found evidence that industry had an important role in driving the discussion at least in knee arthroplasty-a leading field of CAOS-with the ratio between patents and publications increased from approximately 1:10 in 2004 to almost 1:3 in 2014. The adoption of CAOS is largely restrained by economics and ergonomics with sceptics challenging the accuracy and precision of navigation during the early years of CAOS moving to patient functional improvements and long term survivorship. Nevertheless, the future of CAOS remains positive with the prospect of new technologies such as improvements in image-guided surgery, enhanced navigation systems, robotics and artificial intelligence

Automatic tailoring and cloth modelling for animation characters.

The construction of realistic characters has become increasingly important to the production of blockbuster films, TV series and computer games. The outfit of character plays an important role in the application of virtual characters. It is one of the key elements reflects the personality of character. Virtual clothing refers to the process that constructs outfits for virtual characters, and currently, it is widely used in mainly two areas, fashion industry and computer animation. In fashion industry, virtual clothing technology is an effective tool which creates, edits and pre-visualises cloth design patterns efficiently. However, using this method requires lots of tailoring expertises. In computer animation, geometric modelling methods are widely used for cloth modelling due to their simplicity and intuitiveness. However, because of the shortage of tailoring knowledge among animation artists, current existing cloth design patterns can not be used directly by animation artists, and the appearance of cloth depends heavily on the skill of artists. Moreover, geometric modelling methods requires lots of manual operations. This tediousness is worsen by modelling same style cloth for different characters with different body shapes and proportions. This thesis addresses this problem and presents a new virtual clothing method which includes automatic character measuring, automatic cloth pattern adjustment, and cloth patterns assembling. There are two main contributions in this research. Firstly, a geodesic curvature flow based geodesic computation scheme is presented for acquiring length measurements from character. Due to the fast growing demand on usage of high resolution character model in animation production, the increasing number of characters need to be handled simultaneously as well as improving the reusability of 3D model in film production, the efficiency of modelling cloth for multiple high resolution character is very important. In order to improve the efficiency of measuring character for cloth fitting, a fast geodesic algorithm that has linear time complexity with a small bounded error is also presented. Secondly, a cloth pattern adjusting genetic algorithm is developed for automatic cloth fitting and retargeting. For the reason that that body shapes and proportions vary largely in character design, fitting and transferring cloth to a different character is a challenging task. This thesis considers the cloth fitting process as an optimization procedure. It optimizes both the shape and size of each cloth pattern automatically, the integrity, design and size of each cloth pattern are evaluated in order to create 3D cloth for any character with different body shapes and proportions while preserve the original cloth design. By automating the cloth modelling process, it empowers the creativity of animation artists and improves their productivity by allowing them to use a large amount of existing cloth design patterns in fashion industry to create various clothes and to transfer same design cloth to characters with different body shapes and proportions with ease

A Study of Image-based C-arm Tracking Using Minimal Fiducials

Image-based tracking of the c-arm continues to be a critical and challenging problem for many clinical applications due to its widespread use in many computer-assisted procedures that rely upon its accuracy for further planning, registration, and reconstruction tasks. In this thesis, a variety of approaches are presented to improve current c-arm tracking methods and devices for intra-operative procedures. The first approach presents a novel two-dimensional fiducial comprising a set of coplanar conics and an improved single-image pose estimation algorithm that addresses segmentation errors using a mathematical equilibration approach. Simulation results show an improvement in the mean rotation and translation errors by factors of 4 and 1.75, respectively, as a result of using the proposed algorithm. Experiments using real data obtained by imaging a simple precisely machined model consisting of three coplanar ellipses retrieve pose estimates that are in good agreement with those obtained by a ground truth optical tracker. This two-dimensional fiducial can be easily placed under the patient allowing a wide field of view for the motion of the c-arm. The second approach employs learning-based techniques to two-view geometrical theories. A demonstrative algorithm is used to simultaneously tackle matching and segmentation issues of features segmented from pairs of acquired images. The corrected features can then be used to retrieve the epipolar geometry which can ultimately provide pose parameters using a one-dimensional fiducial. The problem of match refinement for epipolar geometry estimation is formulated in a reinforcement-learning framework. Experiments demonstrate the ability to both reject false matches and fix small localization errors in the segmentation of true noisy matches in a minimal number of steps. The third approach presents a feasibility study for an approach that entirely eliminates the use of tracking fiducials. It relies only on preoperative data to initialize a point-based model that is subsequently used to iteratively estimate the pose and the structure of the point-like intraoperative implant using three to six images simultaneously. This method is tested in the framework of prostate brachytherapy in which preoperative data including planned 3-D locations for a large number of point-like implants called seeds is usually available. Simultaneous pose estimation for the c-arm for each image and localization of the seeds is studied in a simulation environment. Results indicate mean reconstruction errors that are less than 1.2 mm for noisy plans of 84 seeds or fewer. These are attained when the 3D mean error introduced to the plan as a result of adding Gaussian noise is less than 3.2 mm

Bio-inspired micro-structural design for CFRP: exploring damage mechanisms of nacre

Carbon-Fibre Reinforced Polymers (CFRP) are widely regarded as the material of choice for many aerospace and automotive applications, where high specific strength and stiffness are required. However, one of the main limitations to further extending the use of such materials is their inherent brittleness, which results in the difficulty to design damage-tolerant lightweight structures. Among the innovative solutions recently proposed to improve the damage tolerance of CFRP, biomimetics provides a valuable source of inspiration. In particular, of the many biological composites, nacre is one that provides a remarkably tough behaviour, due to its discontinuous tiled micro-structure leading to damage-diffusion and crack-deflection mechanisms acting at the micro-scale. In this work, a carbon-fibre/epoxy composite with nacre-inspired tiled micro-structure is firstly designed and synthesised. Analytical and numerical models are developed to identify suitable configurations for the tile geometry with interlocks, leading to tiles of about 600 μm which are then laser-engraved in the laminate plies. In-situ bending tests show how the nacre-like interlocking micro-structure succeeds in diverting cracks and avoiding localised failure, while toughness and heterogeneity of the interface between tiles are identified as key elements to promote further spreading of damage. In order to improve the ductility of the interface, a film-casting technique is developed to deposit extremely thin layers (~13 μm) of poly(lactic acid) (PLA) onto the interface of carbon/epoxy prepregs. Different patterns of PLA texture (including fractals) are explored, with DCB and 4ENF tests showing an increase in interlaminar toughness of about 80% for Mode I and 12% for Mode II. The film-casting method mentioned above is then used to modify the interfaces of the nacre-inspired laminate, by depositing thin PLA patches with fractal shape in between plies. This results in a thin texture of thermoplastic material that toughens the interface without significantly increasing the overall thickness of the laminate. Results show that damage diffusion is considerably enhanced by the tougher and more heterogeneous interface, which succeeds in creating more extensive pull-out of the interlocking tiles. Finally, the interaction between nacre-inspired discontinuous micro-structures and continuous fibre-reinforced layers is analysed. It is shown how continuous layers can be used to trigger unstable failure in the nacre-like material, to act as a crack-propagation barrier, or to change the morphology of damage by promoting a transition from brittle failure to energy-dissipating tile pull-out.Open Acces