InDepth: Real-time Depth Inpainting for Mobile Augmented Reality

Peer reviewe

Online Semantic Labeling of Deformable Tissues for Medical Applications

University of Minnesota Ph.D. dissertation. May 2017. Major: Mechanical Engineering. Advisor: Timothy Kowalewski. 1 computer file (PDF); ix, 133 pages.Surgery remains dangerous, and accurate knowledge of what is presented to the surgeon can be of great importance. One technique to automate this problem is non-rigid tracking of time-of-flight camera scans. This requires accurate sensors and prior information as well as an accurate non-rigid tracking algorithm. This thesis presents an evaluation of four algorithms for tracking and semantic labeling of deformable tissues for medical applications, as well as additional studies on a stretchable flexible smart skin and dynamic 3D bioprinting. The algorithms were developed and tested for this study, and were evaluated in terms of speed and accuracy. The algorithms tested were affine iterative closest point, nested iterative closest point, affine fast point feature histograms, and nested fast point feature histograms. The algorithms were tested against simulated data as well as direct scans. The nested iterative closest point algorithm provided the best balance of speed and accuracy while providing semantic labeling in both simulation as well as using directly scanned data. This shows that fast point feature histograms are not suitable for nonrigid tracking of geometric feature poor human tissues. Secondary experiments were also performed to show that the graphics processing unit provides enough speed to perform iterative closest point algorithms in real-time and that time of flight depth sensing works through an endoscope. Additional research was conducted on related topics, leading to the development of a novel stretchable flexible smart skin sensor and an active 3D bioprinting system for moving human anatomy

Optical dosimetry tools and Monte Carlo based methods for applications in image guided optical therapy in the brain

Purpose: The long-term goal of this research is to determine the feasibility of using near infra-red light to stimulate drug release in metastatic lesions within the brain. In this work, we focused on developing the tools needed to quantify and verify photon fluence distribution in biological tissue. To accomplish this task, an optical dosimetry probe and Monte Carlo based simulation code were fabricated, calibrated and developed to predict light transport in heterogeneous tissue phantoms of the skull and brain. Empirical model (EM) of photon transport using CT images as input were devised to provide real-time calculations capable of being translated to preclinical and clinical applications. Methods and Materials: A GPU based 3D Monte Carlo code was customized to simulate the photon transport within head phantoms consisting of skull bone, white and gray matter with differing laser beam properties, including flat, Gaussian, and super-Gaussian profiles that are converging, parallel, or diverging. From these simulations, the local photon fluence and tissue dosimetric distribution was simulated and validated through the implementation of a novel titanium-based optical dosimetry probe with an isotropic acceptance and 1.5mm diameter. Empirical models (EM) of photon transport were devised and calibrated to MC simulated data to provide 3D fluence and optical dosimetric maps in real-time developed around on a voxel-based convolution technique. Optical transmission studies were performed using human skull bone samples to determine the optical transmission characteristics of heterogeneous bone structures and the effectiveness of the Monte Carlo in simulating this heterogeneity. These tools provide the capability to develop and optimize treatment plans for optimal release of pharmaceuticals to metastatic breast cancer in the brain. Results: At the time of these experiments, the voxel-based CUDA MC code implemented and further developed in this study had not been validated by measurement. A novel optical dosimetry probe was fabricated and calibrated to measure the absolute photon fluence (mW/mm2) in phantoms resembling white matter, gray matter and skull bone and compared to 3D Monte Carlo simulated data. The TiO2-based dosimetry probe was shown to have superior linearity and isotropicity of response to previous Nylon based probes, and was better suited to validate the Monte Carlo using localized 3D measurement (\u3c 25% systematic error for white matter, gray matter and skull bone phantoms along illumination beam axis up to a depth of 2cm in homogeneous tissue and 3.8cm in heterogeneous head phantom). Next, the transport parameters of the empirical algorithm was calibrated using the 3D Monte Carlo and EMs and validated by optical dosimetry probe measurements (with error of 10.1% for White Matter, 45.1% for Gray Matter and 22.1% for Skull Bone phantoms) along illumination beam axis. Conclusions: The design and validation of the Monte Carlo, the optical dosimetry probe and the Empirical algorithm increases the clinical feasibility of optical therapeutic planning to narrow down the complex possibilities of illumination conditions, further compounded by the heterogeneous structure of the brain, such as varying skull thicknesses and densities. Our ultimate goal is to design a fast Monte Carlo based optical therapeutic protocol to treat brain metastasis. The voxelated nature of the MC and EM provides the necessary 3D photon distribution to within 25% error to guide future clinical studies involving optically triggered drug release

Deep Learning for Instrumented Ultrasonic Tracking: From synthetic training data to in vivo application

Instrumented ultrasonic tracking is used to improve needle localisation during ultrasound guidance of minimally-invasive percutaneous procedures. Here, it is implemented with transmitted ultrasound pulses from a clinical ultrasound imaging probe that are detected by a fibre-optic hydrophone integrated into a needle. The detected transmissions are then reconstructed to form the tracking image. Two challenges are considered with the current implementation of ultrasonic tracking. First, tracking transmissions are interleaved with the acquisition of B-mode images and thus, the effective B-mode frame rate is reduced. Second, it is challenging to achieve an accurate localisation of the needle tip when the signal-to-noise ratio is low. To address these challenges, we present a framework based on a convolutional neural network (CNN) to maintain spatial resolution with fewer tracking transmissions and to enhance signal quality. A major component of the framework included the generation of realistic synthetic training data. The trained network was applied to unseen synthetic data and experimental in vivo tracking data. The performance of needle localisation was investigated when reconstruction was performed with fewer (up to eight-fold) tracking transmissions. CNN-based processing of conventional reconstructions showed that the axial and lateral spatial resolution could be improved even with an eight-fold reduction in tracking transmissions. The framework presented in this study will significantly improve the performance of ultrasonic tracking, leading to faster image acquisition rates and increased localisation accuracy

Multi­-Scattering: Computational light transport in turbid media

This thesis presents and describes the development of an online freely accessible software called Multi-Scattering for the computational modeling of light propagation in scattering and absorbing media. The model is based on the use of the Monte Carlo method, where billions of photon packets are being launched and tracked through simulated cubic volumes. The software also includes features for modeling image formation by inserting a virtual collecting lens and a detection matrix which simulate a camera objective and a sensor array respectively. In addition, the Lorenz-Mie theory is integrated to generate the scattering phase functions from spherical particles. The model has been accelerated by means of general-purpose computing on graphics processing units, reducing the computation time by a factor up to 200x in comparison with a single CPU thread. By using four graphic cards on a single computer, the simulation speed increases by a factor of 800x. With an anisotropy factor g= 0.86, the transport path of one billion photons can be computed in 10 seconds for optical depth OD=10 and in 20 minutes for OD=500.The simulations are running from a computer server at Lund University, allowing researchers to login and use it freely without any need for programming skills or specific software/hardware installations. There are countless types of scattering media in which this model can be used to predict photon transport, including medical tissues, blood samples, clouds, smoke, fog, turbid liquids, spray systems, etc. In this thesis, the software has been used for a variety of scattering situations and to simulate photon transport: 1) inside a portion of a human head, 2) within atomizing spray systems, 3) in controlled aqueous dispersion of polystyren spheres, 4) for time-of-flight measurements in intralipid solutions and 5) for Diffuse Correlation Spectroscopy applications.Finally, the numerical results have been validated by rigorously comparing the simulated results with experimental data. The user interface for both setting-up a simulation and displaying the corresponding results is found at: https://multi-scattering.co

Advancements and Breakthroughs in Ultrasound Imaging

Ultrasonic imaging is a powerful diagnostic tool available to medical practitioners, engineers and researchers today. Due to the relative safety, and the non-invasive nature, ultrasonic imaging has become one of the most rapidly advancing technologies. These rapid advances are directly related to the parallel advancements in electronics, computing, and transducer technology together with sophisticated signal processing techniques. This book focuses on state of the art developments in ultrasonic imaging applications and underlying technologies presented by leading practitioners and researchers from many parts of the world

Real-time hybrid cutting with dynamic fluid visualization for virtual surgery

It is widely accepted that a reform in medical teaching must be made to meet today's high volume training requirements. Virtual simulation offers a potential method of providing such trainings and some current medical training simulations integrate haptic and visual feedback to enhance procedure learning. The purpose of this project is to explore the capability of Virtual Reality (VR) technology to develop a training simulator for surgical cutting and bleeding in a general surgery

Proceedings, MSVSCC 2014

Proceedings of the 8th Annual Modeling, Simulation & Visualization Student Capstone Conference held on April 17, 2014 at VMASC in Suffolk, Virginia

Deep Model for Improved Operator Function State Assessment

A deep learning framework is presented for engagement assessment using EEG signals. Deep learning is a recently developed machine learning technique and has been applied to many applications. In this paper, we proposed a deep learning strategy for operator function state (OFS) assessment. Fifteen pilots participated in a flight simulation from Seattle to Chicago. During the four-hour simulation, EEG signals were recorded for each pilot. We labeled 20- minute data as engaged and disengaged to fine-tune the deep network and utilized the remaining vast amount of unlabeled data to initialize the network. The trained deep network was then used to assess if a pilot was engaged during the four-hour simulation