Automating the multimodal analysis of musculoskeletal imaging in the presence of hip implants

In patients treated with hip arthroplasty, the muscular condition and presence of inflammatory reactions are assessed using magnetic resonance imaging (MRI). As MRI lacks contrast for bony structures, computed tomography (CT) is preferred for clinical evaluation of bone tissue and orthopaedic surgical planning. Combining the complementary information of MRI and CT could improve current clinical practice for diagnosis, monitoring and treatment planning. In particular, the different contrast of these modalities could help better quantify the presence of fatty infiltration to characterise muscular condition after hip replacement. In this thesis, I developed automated processing tools for the joint analysis of CT and MR images of patients with hip implants. In order to combine the multimodal information, a novel nonlinear registration algorithm was introduced, which imposes rigidity constraints on bony structures to ensure realistic deformation. I implemented and thoroughly validated a fully automated framework for the multimodal segmentation of healthy and pathological musculoskeletal structures, as well as implants. This framework combines the proposed registration algorithm with tailored image quality enhancement techniques and a multi-atlas-based segmentation approach, providing robustness against the large population anatomical variability and the presence of noise and artefacts in the images. The automation of muscle segmentation enabled the derivation of a measure of fatty infiltration, the Intramuscular Fat Fraction, useful to characterise the presence of muscle atrophy. The proposed imaging biomarker was shown to strongly correlate with the atrophy radiological score currently used in clinical practice. Finally, a preliminary work on multimodal metal artefact reduction, using an unsupervised deep learning strategy, showed promise for improving the postprocessing of CT and MR images heavily corrupted by metal artefact. This work represents a step forward towards the automation of image analysis in hip arthroplasty, supporting and quantitatively informing the decision-making process about patient’s management

A Survey on Deep Learning in Medical Image Analysis

Deep learning algorithms, in particular convolutional networks, have rapidly become a methodology of choice for analyzing medical images. This paper reviews the major deep learning concepts pertinent to medical image analysis and summarizes over 300 contributions to the field, most of which appeared in the last year. We survey the use of deep learning for image classification, object detection, segmentation, registration, and other tasks and provide concise overviews of studies per application area. Open challenges and directions for future research are discussed.Comment: Revised survey includes expanded discussion section and reworked introductory section on common deep architectures. Added missed papers from before Feb 1st 201

Active appearance pyramids for object parametrisation and fitting

Object class representation is one of the key problems in various medical image analysis tasks. We propose a part-based parametric appearance model we refer to as an Active Appearance Pyramid (AAP). The parts are delineated by multi-scale Local Feature Pyramids (LFPs) for superior spatial specificity and distinctiveness. An AAP models the variability within a population with local translations of multi-scale parts and linear appearance variations of the assembly of the parts. It can fit and represent new instances by adjusting the shape and appearance parameters. The fitting process uses a two-step iterative strategy: local landmark searching followed by shape regularisation. We present a simultaneous local feature searching and appearance fitting algorithm based on the weighted Lucas and Kanade method. A shape regulariser is derived to calculate the maximum likelihood shape with respect to the prior and multiple landmark candidates from multi-scale LFPs, with a compact closed-form solution. We apply the 2D AAP on the modelling of variability in patients with lumbar spinal stenosis (LSS) and validate its performance on 200 studies consisting of routine axial and sagittal MRI scans. Intervertebral sagittal and parasagittal cross-sections are typically used for the diagnosis of LSS, we therefore build three AAPs on L3/4, L4/5 and L5/S1 axial cross-sections and three on parasagittal slices. Experiments show significant improvement in convergence range, robustness to local minima and segmentation precision compared with Constrained Local Models (CLMs), Active Shape Models (ASMs) and Active Appearance Models (AAMs), as well as superior performance in appearance reconstruction compared with AAMs. We also validate the performance on 3D CT volumes of hip joints from 38 studies. Compared to AAMs, AAPs achieve a higher segmentation and reconstruction precision. Moreover, AAPs have a significant improvement in efficiency, consuming about half the memory and less than 10% of the training time and 15% of the testing time

Towards a framework for multi class statistical modelling of shape, intensity, and kinematics in medical images

Statistical modelling has become a ubiquitous tool for analysing of morphological variation of bone structures in medical images. For radiological images, the shape, relative pose between the bone structures and the intensity distribution are key features often modelled separately. A wide range of research has reported methods that incorporate these features as priors for machine learning purposes. Statistical shape, appearance (intensity profile in images) and pose models are popular priors to explain variability across a sample population of rigid structures. However, a principled and robust way to combine shape, pose and intensity features has been elusive for four main reasons: 1) heterogeneity of the data (data with linear and non-linear natural variation across features); 2) sub-optimal representation of three-dimensional Euclidean motion; 3) artificial discretization of the models; and 4) lack of an efficient transfer learning process to project observations into the latent space. This work proposes a novel statistical modelling framework for multiple bone structures. The framework provides a latent space embedding shape, pose and intensity in a continuous domain allowing for new approaches to skeletal joint analysis from medical images. First, a robust registration method for multi-volumetric shapes is described. Both sampling and parametric based registration algorithms are proposed, which allow the establishment of dense correspondence across volumetric shapes (such as tetrahedral meshes) while preserving the spatial relationship between them. Next, the framework for developing statistical shape-kinematics models from in-correspondence multi-volumetric shapes embedding image intensity distribution, is presented. The framework incorporates principal geodesic analysis and a non-linear metric for modelling the spatial orientation of the structures. More importantly, as all the features are in a joint statistical space and in a continuous domain; this permits on-demand marginalisation to a region or feature of interest without training separate models. Thereafter, an automated prediction of the structures in images is facilitated by a model-fitting method leveraging the models as priors in a Markov chain Monte Carlo approach. The framework is validated using controlled experimental data and the results demonstrate superior performance in comparison with state-of-the-art methods. Finally, the application of the framework for analysing computed tomography images is presented. The analyses include estimation of shape, kinematic and intensity profiles of bone structures in the shoulder and hip joints. For both these datasets, the framework is demonstrated for segmentation, registration and reconstruction, including the recovery of patient-specific intensity profile. The presented framework realises a new paradigm in modelling multi-object shape structures, allowing for probabilistic modelling of not only shape, but also relative pose and intensity as well as the correlations that exist between them. Future work will aim to optimise the framework for clinical use in medical image analysis

Deformable Multisurface Segmentation of the Spine for Orthopedic Surgery Planning and Simulation

Purpose: We describe a shape-aware multisurface simplex deformable model for the segmentation of healthy as well as pathological lumbar spine in medical image data. Approach: This model provides an accurate and robust segmentation scheme for the identification of intervertebral disc pathologies to enable the minimally supervised planning and patient-specific simulation of spine surgery, in a manner that combines multisurface and shape statistics-based variants of the deformable simplex model. Statistical shape variation within the dataset has been captured by application of principal component analysis and incorporated during the segmentation process to refine results. In the case where shape statistics hinder detection of the pathological region, user assistance is allowed to disable the prior shape influence during deformation. Results: Results demonstrate validation against user-assisted expert segmentation, showing excellent boundary agreement and prevention of spatial overlap between neighboring surfaces. This section also plots the characteristics of the statistical shape model, such as compactness, generalizability and specificity, as a function of the number of modes used to represent the family of shapes. Final results demonstrate a proof-of-concept deformation application based on the open-source surgery simulation Simulation Open Framework Architecture toolkit. Conclusions: To summarize, we present a deformable multisurface model that embeds a shape statistics force, with applications to surgery planning and simulation

Analysis, Segmentation and Prediction of Knee Cartilage using Statistical Shape Models

Osteoarthritis (OA) of the knee is one of the leading causes of chronic disability (along with the hip). Due to rising healthcare costs associated with OA, it is important to fully understand the disease and how it progresses in the knee. One symptom of knee OA is the degeneration of cartilage in the articulating knee. The cartilage pad plays a major role in painting the biomechanical picture of the knee. This work attempts to quantify the cartilage thickness of healthy male and female knees using statistical shape models (SSMs) for a deep knee bend activity. Additionally, novel cartilage segmentation from magnetic resonance imaging (MRI) and estimation algorithms from computer tomography (CT) or x-rays are proposed to facilitate the efficient development and accurate analysis of future treatments related to the knee. Cartilage morphology results suggest distinct patterns of wear in varus, valgus, and neutral degenerative knees, and examination of contact regions during the deep knee bend activity further emphasizes these patterns. Segmentation results were achieved that were comparable if not of higher quality than existing state-of-the-art techniques for both femoral and tibial cartilage. Likewise, using the point correspondence properties of SSMs, estimation of articulating cartilage was effective in healthy and degenerative knees. In conclusion, this work provides novel, clinically relevant morphological data to compute segmentation and estimate new data in such a way to potentially contribute to improving results and efficiency in evaluation of the femorotibial cartilage layer

Visualizing and Predicting the Effects of Rheumatoid Arthritis on Hands

This dissertation was inspired by difficult decisions patients of chronic diseases have to make about about treatment options in light of uncertainty. We look at rheumatoid arthritis (RA), a chronic, autoimmune disease that primarily affects the synovial joints of the hands and causes pain and deformities. In this work, we focus on several parts of a computer-based decision tool that patients can interact with using gestures, ask questions about the disease, and visualize possible futures. We propose a hand gesture based interaction method that is easily setup in a doctor\u27s office and can be trained using a custom set of gestures that are least painful. Our system is versatile and can be used for operations like simple selections to navigating a 3D world. We propose a point distribution model (PDM) that is capable of modeling hand deformities that occur due to RA and a generalized fitting method for use on radiographs of hands. Using our shape model, we show novel visualization of disease progression. Using expertly staged radiographs, we propose a novel distance metric learning and embedding technique that can be used to automatically stage an unlabeled radiograph. Given a large set of expertly labeled radiographs, our data-driven approach can be used to extract different modes of deformation specific to a disease

Quantitative Magnetic Resonance Imaging and Analysis of Articular Cartilage and Osteoarthritis

MRI plays an important role in the continuing search for a sensitive osteoarthritis (OA) imaging biomarker able to detect early, pre-morphological alterations in cartilage composition. Determining the compositional recovery pattern of cartilage following acute joint loading could potentially present a more sensitive biomarker for defining cartilage health [1]. However, only a limited amount of studies have assessed both the immediate effect of joint loading on cartilage, as well as its post-loading recovery. In addition, when assessing the compositional responses of cartilage to joint loading, previous studies usually did not incorporate the measurement error of the used quantitative MRI technique into their analysis. Therefore, an uncertainty persists whether or not compositional MRI techniques are sensitive enough to measure changes in water and macromolecular content of cartilage, or if previous studies were merely measuring noise. Consequently, an objective of this thesis is to increase our understanding of and reliability in quantitative T2 and T1ρ relaxation time mapping to detect compositional responses of cartilage following a joint loading activity. Furthermore, to obtain the quantitative morphological and compositional measures of cartilage, detailed region-specific delineation of cartilage is required. This delineation (or segmentation) of cartilage is laborious and time-consuming as it is usually performed manually by an expert observer. Many new advances in image analysis, particularly those in convolutional neural networks (CNNs) and deep learning, have enabled a time-efficient semi- or fully-automated alternative to this process [2, 3]. This thesis explores the utility of deep CNNs generated segmentations for accurate surface-based analysis of cartilage morphology and composition from knee MRIs as well as of cortical bone thickness from knee CTs. Chapter 1 will provide an introduction into the structure and biomechanics of articular cartilage and the role of MRI in imaging the degenerative joint disorder, osteoarthritis as well as the effects of different joint loading activities on cartilage morphology and composition. Chapter 2 explains the principle of MRI and the pulse sequences used in the following chapter for the morphometric and compositional assessment of articular cartilage. Chapter 3 describes the use of 3D Cartilage Surface Mapping (3D-CaSM) [3] to assess variations in cartilage T1ρ and T2 relaxation times of young, healthy participants following a mild, unilateral stepping activity. By evaluating and incorporating the intrasessional repeatability of the T1ρ and T2 mapping techniques, I aim to highlight those cartilage areas experiencing exercise-induced compositional changes greater than measurement error. A significant amount of time is needed to manually segment the regions-of-interest required to perform the 3D-CaSM used in Chapter 3. Therefore, in Chapter 4, I assessed the use of deep convolutional neural networks for automating the segmentation process for multiple knee joint tissues simultaneous and increase the time-efficiency for evaluating knee MR datasets. I evaluated the use of a conditional Generative Adversarial Network (cGAN) as a potentially improved method for automated segmentation compared to the widely used convolutional neural network, U-Net. In Chapter 5 I combined the 3D-CaSM and automated segmentation methods presented in Chapters 3 and 4, respectively to assess the use of fully automatic segmentations of femoral and tibial bone-cartilage structures for accurate surface-based analysis of cartilage morphology and composition on knee MR images. This was performed on publicly available data from the Osteoarthritis Initiative, a multicentre observational study with expert manual segmentations provided by the Zuse Institute in Berlin. Chapter 6 describes an automated pipeline for subchondral cortical bone thickness mapping from knee CT data. I developed a method of using automated segmentations of articular cartilage and bone from knee MRI data to determine the periarticular bone surface which is covered by cartilage. This surface was then used to perform cortical bone thickness measurements on corresponding CT data. I validated this pipeline using data from the EU-funded, multi-centre observational study called Applied Private-Public partneRship enabling OsteoArthritis Clinical Headway (APPROACH). Chapter 7 summarises the main conclusions and contributions of the works presented in this thesis as well as providing directions for future work.PhD Studentship funded by GlaxoSmithKlin