Multi-Surface Simplex Spine Segmentation for Spine Surgery Simulation and Planning

This research proposes to develop a knowledge-based multi-surface simplex deformable model for segmentation of healthy as well as pathological lumbar spine data. It aims to provide a more accurate and robust segmentation scheme for identification of intervertebral disc pathologies to assist with spine surgery planning. A robust technique that combines multi-surface and shape statistics-aware variants of the deformable simplex model is presented. Statistical shape variation within the dataset has been captured by application of principal component analysis and incorporated during the segmentation process to refine results. In the case where shape statistics hinder detection of the pathological region, user-assistance is allowed to disable the prior shape influence during deformation. Results have been validated against user-assisted expert segmentation

Geometric morphometrics for 3D dense surface correspondence: population comparisons of shoulder bone morphology

Background: Comparisons in morphological shape/form across population groups could provide population differences that might assist in making decisions on diagnosis and prognosis by the clinician. Geometric morphometrics (GM) is one of the fields that help to provide such population comparisons. In medical imaging and related disciplines, GM is commonly done using annotated landmarks or distances measured from 3D surfaces (consisting of triangular meshes). However, these landmarks may not be sufficient to describe the complete shape. This project aimed to develop GM for analysis that consider all vertices in the triangular mesh as landmarks. The developed methods were applied to South African and Swiss shoulder bones (scapula and humerus) to analyse morphological differences. Methods: The developed pipeline required first establishing correspondence across the datasets through a registration process. Gaussian process fitting was chosen to perform the registration since it is considered state-of-the-art. Secondly, a novel method for automatic identification of vertices or areas encoding the most shape/form variation was developed. Thirdly, a principal component analysis (PCA) that addressed the high dimensionality and lower sample size (HDLSS) phenomenon was adopted and applied to the dense correspondence data. This approach allowed for the stabilisation of the distribution of the data in low-dimensional form/shape space. Lastly, appropriate statistical tests were developed for population comparisons of the shoulder bones when dealing with HDLSS data in both form and shape space. Results: When the mesh-based GM analysis approach was applied to the training datasets (South African and Swiss shoulder bones), it was found that the anterior glenoid which is often the site of the shoulder dislocation is the most varied area of the glenoid. This has implications for diagnosis and provides knowledge for prosthesis design. The distribution of the data in the modified PCA space was shown to converge to a stable distribution when more vertices/landmarks are used for the analysis. South African and Swiss datasets were shown to be more distinguishable in a low-dimensional space when considering form rather than shape. It was found that left and right South African scapula bones are significantly different in terms of shape. Discussion: In general, it was observed that the two populations means can be significantly different in shape but not in form. An improved understanding of these observed shape and form differences has utility for shoulder arthroplasty prosthesis design and may also be useful for orthopaedic surgeons during surgical preoperative planning

A deep learning algorithm for contour detection in synthetic 2D biplanar X-ray images of the scapula: towards improved 3D reconstruction of the scapula

Three-dimensional (3D) reconstruction from X-ray images using statistical shape models (SSM) provides a cost-effective way of increasing the diagnostic utility of two-dimensional (2D) X-ray images, especially in low-resource settings. The landmark-constrained model fitting approach is one way to obtain patient-specific models from a statistical model. This approach requires an accurate selection of corresponding features, usually landmarks, from the bi-planar X-ray images. However, X-ray images are 2D representations of 3D anatomy with super-positioned structures, which confounds this approach. The literature shows that detection and use of contours to locate corresponding landmarks within biplanar X-ray images can address this limitation. The aim of this research project was to train and validate a deep learning algorithm for detection the contour of a scapula in synthetic 2D bi-planar Xray images. Synthetic bi-planar X-ray images were obtained from scapula mesh samples with annotated landmarks generated from a validated SSM obtained from the Division of Biomedical Engineering, University of Cape Town. This was followed by the training of two convolutional neural network models as the first objective of the project; the first model was trained to predict the lateral (LAT) scapula image given the anterior-posterior (AP) image. The second model was trained to predict the AP image given the LAT image. The trained models had an average Dice coefficient value of 0.926 and 0.964 for the predicted LAT and AP images, respectively. However, the trained models did not generalise to the segmented real X-ray images of the scapula. The second objective was to perform landmark-constrained model fitting using the corresponding landmarks embedded in the predicted images. To achieve this objective, the 2D landmark locations were transformed into 3D coordinates using the direct linear transformation. The 3D point localization yielded average errors of (0.35, 0.64, 0.72) mm in the X, Y and Z directions, respectively, and a combined coordinate error of 1.16 mm. The reconstructed landmarks were used to reconstruct meshes that had average surface-to-surface distances of 3.22 mm and 1.72 mm for 3 and 6 landmarks, respectively. The third objective was to reconstruct the scapula mesh using matching points on the scapula contour in the bi-planar images. The average surface-to-surface distances of the reconstructed meshes with 8 matching contour points and 6 corresponding landmarks of the same meshes were 1.40 and 1.91 mm, respectively. In summary, the deep learning models were able to learn the mapping between the bi-planar images of the scapula. Increasing the number of corresponding landmarks from the bi-planar images resulted into better 3D reconstructions. However, obtaining these corresponding landmarks was non-trivial, necessitating the use of matching points selected from the scapulae contours. The results from the latter approach signal a need to explore contour matching methods to obtain more corresponding points in order to improve the scapula 3D reconstruction using landmark-constrained model fitting

Statistical Shape and Intensity Modeling of the Shoulder

Anatomical variability in the shoulder is inherently present and can influence healthy and pathologic biomechanics and ultimately clinical decision-making. Characterizing variation in bony morphology and material properties in the population can support treatment and specifically the design, via shape and sizing, of shoulder implants. Total Shoulder Arthroplasty (TSA) is the treatment of choice for glenohumeral osteoarthritis as well as bone fracture. Complications and poor outcomes in TSA are generally influenced by the inability of the implant to replicate the natural joint biomechanics and by the bone quality around the fixation features. For this reason, knowledge of bony morphology and mechanical properties can support optimal implant design and sizing, and thus improve TSA results. Statistical shape and intensity modeling is a powerful tool to represent the shape and mechanical properties variation in a training set. Accordingly, the objectives of this thesis were: 1) to develop a statistical shape model (SSM) of the proximal humeral cortical and cancellous bone; 2) to develop an SSM and a statistical intensity model (SIM) of the scapular bone. A training set of 85 humeri and 53 scapulae were reconstructed from CT scans and registered to common templates. Principal Component Analysis (PCA) was applied to the registered geometries to quantify morphological and bone properties variation in the population. For both the humerus and the scapula SSM, the first mode of variation accounted for most of the variation and described scaling. Subsequent modes described changes in the scapular plate, acromion process and scapular notch for the scapula, and in the neck angle, head inclination, greater and lesser tubercles for the humerus. Variation in cortical thickness of the humeral diaphysis was largely independent of size and statistically significant differences with ethnicity were noted. Asian subjects showed higher humeral cortical thickness with respect to Caucasians, regardless of gender. The first mode of variation in the scapular SIM described scaling in material properties distribution, with higher bone density located centrally and anteriorly in the glenoid region. The bone property maps developed for the scapular training set realistically captured inter-subject variability and they represent a valuable tool to assess fixation features and screw location and trajectories for TSA glenoid component. The SSMs and SIM developed in this thesis represent a useful infrastructure to support population-based evaluations and assess possible anatomical differences with gender and ethnicity, SSM and SIM can also provide anatomical relationship in support of implant design and sizing

Towards a framework for multi class statistical modelling of shape, intensity, and kinematics in medical images

Statistical modelling has become a ubiquitous tool for analysing of morphological variation of bone structures in medical images. For radiological images, the shape, relative pose between the bone structures and the intensity distribution are key features often modelled separately. A wide range of research has reported methods that incorporate these features as priors for machine learning purposes. Statistical shape, appearance (intensity profile in images) and pose models are popular priors to explain variability across a sample population of rigid structures. However, a principled and robust way to combine shape, pose and intensity features has been elusive for four main reasons: 1) heterogeneity of the data (data with linear and non-linear natural variation across features); 2) sub-optimal representation of three-dimensional Euclidean motion; 3) artificial discretization of the models; and 4) lack of an efficient transfer learning process to project observations into the latent space. This work proposes a novel statistical modelling framework for multiple bone structures. The framework provides a latent space embedding shape, pose and intensity in a continuous domain allowing for new approaches to skeletal joint analysis from medical images. First, a robust registration method for multi-volumetric shapes is described. Both sampling and parametric based registration algorithms are proposed, which allow the establishment of dense correspondence across volumetric shapes (such as tetrahedral meshes) while preserving the spatial relationship between them. Next, the framework for developing statistical shape-kinematics models from in-correspondence multi-volumetric shapes embedding image intensity distribution, is presented. The framework incorporates principal geodesic analysis and a non-linear metric for modelling the spatial orientation of the structures. More importantly, as all the features are in a joint statistical space and in a continuous domain; this permits on-demand marginalisation to a region or feature of interest without training separate models. Thereafter, an automated prediction of the structures in images is facilitated by a model-fitting method leveraging the models as priors in a Markov chain Monte Carlo approach. The framework is validated using controlled experimental data and the results demonstrate superior performance in comparison with state-of-the-art methods. Finally, the application of the framework for analysing computed tomography images is presented. The analyses include estimation of shape, kinematic and intensity profiles of bone structures in the shoulder and hip joints. For both these datasets, the framework is demonstrated for segmentation, registration and reconstruction, including the recovery of patient-specific intensity profile. The presented framework realises a new paradigm in modelling multi-object shape structures, allowing for probabilistic modelling of not only shape, but also relative pose and intensity as well as the correlations that exist between them. Future work will aim to optimise the framework for clinical use in medical image analysis

Automatic Image Segmentation of Healthy and Atelectatic Lungs in Computed Tomography

Computed tomography (CT) has become a standard in pulmonary imaging which allows the analysis of diseases like lung nodules, emphysema and embolism. The improved spatial and temporal resolution involves a dramatic increase in the amount of data that has to be stored and processed. This has motivated the development of computer aided diagnostics (CAD) systems that have released the physician from the tedious task of manually delineating the boundary of the structures of interest from such a large number of images, a pre-processing step known as image segmentation. Apart from being impractical, the manual segmentation is prone to high intra and inter observer subjectiveness. Automatic segmentation of the lungs with atelectasis poses a challenge because in CT images they have similar texture and gray level as the surrounding tissue. Consequently, the available graphical information is not sufficient to distinguish the boundary of the lung. The present work aims to close the existing gap left by the segmentation of atelectatic lungs in volume CT data. A-priori knowledge of anatomical information plays a key role in the achievement of this goal

Automated Distinct Bone Segmentation from Computed Tomography Images using Deep Learning

Large-scale CT scans are frequently performed for forensic and diagnostic purposes, to plan and direct surgical procedures, and to track the development of bone-related diseases. This often involves radiologists who have to annotate bones manually or in a semi-automatic way, which is a time consuming task. Their annotation workload can be reduced by automated segmentation and detection of individual bones. This automation of distinct bone segmentation not only has the potential to accelerate current workflows but also opens up new possibilities for processing and presenting medical data for planning, navigation, and education. In this thesis, we explored the use of deep learning for automating the segmentation of all individual bones within an upper-body CT scan. To do so, we had to find a network architec- ture that provides a good trade-off between the problem’s high computational demands and the results’ accuracy. After finding a baseline method and having enlarged the dataset, we set out to eliminate the most prevalent types of error. To do so, we introduced an novel method called binary-prediction-enhanced multi-class (BEM) inference, separating the task into two: Distin- guishing bone from non-bone is conducted separately from identifying the individual bones. Both predictions are then merged, which leads to superior results. Another type of error is tack- led by our developed architecture, the Sneaky-Net, which receives additional inputs with larger fields of view but at a smaller resolution. We can thus sneak more extensive areas of the input into the network while keeping the growth of additional pixels in check. Overall, we present a deep-learning-based method that reliably segments most of the over one hundred distinct bones present in upper-body CT scans in an end-to-end trained matter quickly enough to be used in interactive software. Our algorithm has been included in our groups virtual reality medical image visualisation software SpectoVR with the plan to be used as one of the puzzle piece in surgical planning and navigation, as well as in the education of future doctors

Toward adaptive radiotherapy

Intensity Modulated Radiotherapy (IMRT) and proton therapy are the state-of-art external radiotherapy modalities. To make the most of such precise delivery, accurate knowledge of the patient anatomy and biology during treatment is necessary, as unaccounted variations can compromise the outcome of the treatment. Treatment modification to account for deviations from the planning stage is a framework known as adaptive radiotherapy (ART). To fully utilise the information extracted from different modalities and/or at different time-points it is required to accurately align the imaging data. In this work the feasibility of cone-beam computed tomography (CBCT) and deformable image registration (DIR) for ART was evaluated in the context of head and neck (HN) and lung malignancies, and for IMRT and proton therapy applications. This included the geometric validation of deformations for multiple DIR algorithms, estimating the uncertainty in dose recalculation of a CBCT-based deformed CT (dCT), and the uncertainty in dose summation resulting from the properties of the underlying deformations. The dCT method was shown to be a good interim solution to repeat CT and a superior alternative to simpler direct usage of CBCT for dose calculation; proton therapy treatments were more sensitive to registration errors than IMRT. The ability to co-register multimodal and multitemporal data of the HN was also explored; the results found were promising and the limitations of current algorithms and data acquisition protocols were identified. The use of novel artificial cancer masses as a novel platform for the study of imaging during radiotherapy was explored in this study. The artificial cancer mass model was extended to generate magnetic resonance imaging (MRI)-friendly samples. The tumoroids were imageable in standard T1 and T2 MRI acquisitions, and the relaxometric properties were measured. The main limitation of the current tumour model was the poor reproducibility and controllability of the properties of the samples

Development of procedures for the design, optimization and manufacturing of customized orthopaedic and trauma implants: Geometrical/anatomical modelling from 3D medical imaging

Tese de Doutoramento (Programa Doutoral em Engenharia Biomédica)The introduction of imaging techniques in 1970 is one of the most relevant historical milestones in modern medicine. Medical imaging techniques have dramatically changed our understanding of the Human anatomy and physiology. The ability to non-invasively extract visual information allowed, not only the three-dimensional representation of the internal organs and musculo-skeletal system, but also the simulation of surgical procedures, the execution of computer aided surgeries, the development of more accurate biomechanical models, the development of custom-made implants, among others. The combination of the most advanced medical imaging systems with the most advanced CAD and CAM techniques, may allow the development of custom-made implants that meet patient-speci c traits. The geometrical and functional optimization of these devices may increase implant life-expectancy, especially in patients with marked deviations from the anatomical standards. In the implant customization protocol from medical image data, there are several steps that need to be followed in a sequential way, namely: Medical Image Processing and Recovering; Accurate Image Segmentation and 3D Surface Model Generation; Geometrical Customization based on CAD and CAE techniques; FEA Optimization of the Implant Geometry; and Manufacturing using CAD-CAM Technologies. This work aims to develop the necessary procedures for custom implant development from medical image data. This includes the extraction of highly accurate three-dimensional representation of the musculo-skeletal system from the Computed Tomography imaging, and the development of customized implants, given the speci c requirements of the target anatomy, and the applicable best practices found in the literature. A two-step segmentation protocol is proposed. In the rst step the region of interest is pre-segmented in order to obtain a good approximation to the desired geometry. Next, a fully automatic segmentation re nement is applied to obtain a more accurate representation of the target domain. The re nement step is composed by several sub-steps, more precisely, the recovery of the original image, considering the limiting resolution of the imaging system; image cropping; image interpolation; and segmentation re nement over the up-sampled domain. Highly accurate segmentations of the target domain were obtained with the proposed pipeline. The limiting factor to the accurate description of the domain accuracy is the image acquisition process, rather the following image processing, segmentation and surface meshing steps. The new segmentation pipeline was used in the development of three tailor-made implants, namely, a tibial nailing system, a mandibular implant, and a Total Hip Replacement system. Implants optimization is carried with Finite Element Analysis, considering the critical loading conditions that may be applied to each implant in working conditions. The new tibial nailing system is able of sustaining critical loads without implant failure; the new mandibular endoprosthesis that allows the recovery of the natural stress and strain elds observed in intact mandibles; and the Total Hip Replacement system that showed comparable strain shielding levels as commercially available stems. In summary, in the present thesis the necessary procedures for custom implant design are investigated, and new algorithms proposed. The guidelines for the characterization of the image acquisition, image processing, image segmentation and 3D reconstruction are presented and discussed. This new image processing pipeline is applied and validated in the development of the three abovementioned customized implants, for di erent medical applications and that satisfy speci c anatomical needs.Um dos principais marcos da história moderna da medicina e a introdução da imagem médica, em meados da década de 1970. As tecnologias de imagem permitiram aumentar e potenciar o nosso conhecimento acerca da anatomia e fisiologia do corpo Humano. A capacidade de obter informação imagiológica de forma não invasiva permitiu, não são a representação tridimensional de órgãos e do sistema músculo-esquelético, mas também a simulação de procedimentos cirúrgicos, a realização de cirurgias assistidas por computador, a criação de modelos biomecânicos mais realistas, a criação de implantes personalizados, entre outros. A conjugação dos sistemas mais avançados de imagem medica com as técnicas mais avançadas de modelação e maquinagem, pode permitir o desenvolvimento de implantes personalizados mais otimizados, que vão de encontro as especificidades de cada paciente. Por sua vez, a otimização geométrica e biomecânica destes dispositivos pode permitir, quer o aumento da sua longevidade, quer o tratamento de pessoas com estruturas anatómicas que se afastam dos padrões normais. O processo de modelação de implantes a partir da imagem medica passa por um conjunto de procedimentos a adotar, sequencialmente, ate ao produto final, a saber: Processamento e Recuperação de Imagem; Segmentação de Imagem e Reconstrução tridimensional da Região de Interesse; Modelação Geométrica do Implante; Simulação Numérica para a Otimização da Geometria; a Maquinagem do Implante. Este trabalho visa o desenvolvimento dos procedimentos necessários para a criação de implantes personalizados a partir da imagem medica, englobando a extração de modelos ósseos geométricos rigorosos a partir de imagens de Tomografia Computorizada e, a partir desses modelos, desenvolver implantes personalizados baseados nas melhores praticas existentes na literatura e que satisfaçam as especificidades da anatomia do paciente. Assim, apresenta-se e discute-se um novo procedimento de segmentação em dois passos. No primeiro e feita uma pre-segmentação que visa obter uma aproximação iniciala região de interesse. De seguida, um procedimento de refinamento da segmentação totalmente automático e aplicada a segmentação inicial para obter uma descrição mais precisa do domínio de interesse. O processo de refinamento da segmentação e constituído por vários procedimentos, designadamente: recuperação da imagem original, tendo em consideração a resolução limitante do sistema de imagem; o recorte da imagem na vizinhança da região pre-segmentada; a interpolação da região de interesse; e o refinamento da segmentação aplicando a técnica de segmentação Level-Sets sobre o domínio interpolado. O procedimento de segmentação permitiu extrair modelos extremamente precisos a partir da informação imagiológica. Os resultados revelam que o fator limitante a descrição do domínio e o processo de aquisição de imagem, em detrimento dos diversos passos de processamento subsequentes. O novo protocolo de segmentação foi utilizado no desenvolvimento de três implantes personalizados, a saber: um sistema de fixação interna para a tíbia; um implante mandibular; e um sistema para a Reconstrução Total da articulação da Anca. A otimização do comportamento mecânico dos implantes foi feita utilizado o Método dos Elementos Finitos, tendo em conta os carregamentos críticos a que estes podem estar sujeitos durante a sua vida útil. O sistema de fixação interna para a tíbia e capaz de suportar os carregamentos críticos, sem que a sua integridade mecânica seja comprometida; o implante mandibular permite recuperar os campos de tensão e deformação observados em mandíbulas intactas; e a Prótese Total da Anca apresenta níveis de strain shielding ao longo do fémur proximal comparáveis com os níveis observados em dispositivos comercialmente disponíveis. Em suma, nesta tese de Doutoramento são investigados e propostos novos procedimentos para o projeto de implantes feitos por medida. São apresentadas e discutidas as linhas orientadoras para a caracterização precisa do sistema de aquisição de imagem, para o processamento de imagem, para a segmentação, e para a reconstrução 3D das estruturas anatómicas a partir da imagem medica. Este conjunto de linhas orientadoras é aplicado e validado no desenvolvimento de três implantes personalizados, citados anteriormente, para aplicações médicas distintas e que satisfazem as necessidades anatómicas específicas de cada paciente.Fundação para a Ciência e Tecnologia (FCT

Lifelong Learning in the Clinical Open World

Despite mounting evidence that data drift causes deep learning models to deteriorate over time, the majority of medical imaging research is developed for - and evaluated on - static close-world environments. There have been exciting advances in the automatic detection and segmentation of diagnostically-relevant findings. Yet the few studies that attempt to validate their performance in actual clinics are met with disappointing results and little utility as perceived by healthcare professionals. This is largely due to the many factors that introduce shifts in medical image data distribution, from changes in the acquisition practices to naturally occurring variations in the patient population and disease manifestation. If we truly wish to leverage deep learning technologies to alleviate the workload of clinicians and drive forward the democratization of health care, we must move away from close-world assumptions and start designing systems for the dynamic open world. This entails, first, the establishment of reliable quality assurance mechanisms with methods from the fields of uncertainty estimation, out-of-distribution detection, and domain-aware prediction appraisal. Part I of the thesis summarizes my contributions to this area. I first propose two approaches that identify outliers by monitoring a self-supervised objective or by quantifying the distance to training samples in a low-dimensional latent space. I then explore how to maximize the diversity among members of a deep ensemble for improved calibration and robustness; and present a lightweight method to detect low-quality lung lesion segmentation masks using domain knowledge. Of course, detecting failures is only the first step. We ideally want to train models that are reliable in the open world for a large portion of the data. Out-of-distribution generalization and domain adaptation may increase robustness, but only to a certain extent. As time goes on, models can only maintain acceptable performance if they continue learning with newly acquired cases that reflect changes in the data distribution. The goal of continual learning is to adapt to changes in the environment without forgetting previous knowledge. One practical strategy to approach this is expansion, whereby multiple parametrizations of the model are trained and the most appropriate one is selected during inference. In the second part of the thesis, I present two expansion-based methods that do not rely on information regarding when or how the data distribution changes. Even when appropriate mechanisms are in place to fail safely and accumulate knowledge over time, this will only translate to clinical usage insofar as the regulatory framework allows it. Current regulations in the USA and European Union only authorize locked systems that do not learn post-deployment. Fortunately, regulatory bodies are noting the need for a modern lifecycle regulatory approach. I review these efforts, along with other practical aspects of developing systems that learn through their lifecycle, in the third part of the thesis. We are finally at a stage where healthcare professionals and regulators are embracing deep learning. The number of commercially available diagnostic radiology systems is also quickly rising. This opens up our chance - and responsibility - to show that these systems can be safe and effective throughout their lifespan