A probabilistic deep learning model of inter-fraction anatomical variations in radiotherapy

In radiotherapy, the internal movement of organs between treatment sessions causes errors in the final radiation dose delivery. Motion models can be used to simulate motion patterns and assess anatomical robustness before delivery. Traditionally, such models are based on principal component analysis (PCA) and are either patient-specific (requiring several scans per patient) or population-based, applying the same deformations to all patients. We present a hybrid approach which, based on population data, allows to predict patient-specific inter-fraction variations for an individual patient. We propose a deep learning probabilistic framework that generates deformation vector fields (DVFs) warping a patient's planning computed tomography (CT) into possible patient-specific anatomies. This daily anatomy model (DAM) uses few random variables capturing groups of correlated movements. Given a new planning CT, DAM estimates the joint distribution over the variables, with each sample from the distribution corresponding to a different deformation. We train our model using dataset of 312 CT pairs from 38 prostate cancer patients. For 2 additional patients (22 CTs), we compute the contour overlap between real and generated images, and compare the sampled and ground truth distributions of volume and center of mass changes. With a DICE score of 0.86 and a distance between prostate contours of 1.09 mm, DAM matches and improves upon PCA-based models. The distribution overlap further indicates that DAM's sampled movements match the range and frequency of clinically observed daily changes on repeat CTs. Conditioned only on a planning CT and contours of a new patient without any pre-processing, DAM can accurately predict CTs seen during following treatment sessions, which can be used for anatomically robust treatment planning and robustness evaluation against inter-fraction anatomical changes

Le recalage robuste d’images médicales et la modélisation du mouvement basée sur l’apprentissage profond

This thesis presents new computational tools for quantifying deformations and motion of anatomical structures from medical images as required by a large variety of clinical applications. Generic deformable registration tools are presented that enable deformation analysis useful for improving diagnosis, prognosis and therapy guidance. These tools were built by combining state-of-the-art medical image analysis methods with cutting-edge machine learning methods.First, we focus on difficult inter-subject registration problems. By learning from given deformation examples, we propose a novel agent-based optimization scheme inspired by deep reinforcement learning where a statistical deformation model is explored in a trial-and-error fashion showing improved registration accuracy. Second, we develop a diffeomorphic deformation model that allows for accurate multiscale registration and deformation analysis by learning a low-dimensional representation of intra-subject deformations. The unsupervised method uses a latent variable model in form of a conditional variational autoencoder (CVAE) for learning a probabilistic deformation encoding that is useful for the simulation, classification and comparison of deformations.Third, we propose a probabilistic motion model derived from image sequences of moving organs. This generative model embeds motion in a structured latent space, the motion matrix, which enables the consistent tracking of structures and various analysis tasks. For instance, it leads to the simulation and interpolation of realistic motion patterns allowing for faster data acquisition and data augmentation.Finally, we demonstrate the importance of the developed tools in a clinical application where the motion model is used for disease prognosis and therapy planning. It is shown that the survival risk for heart failure patients can be predicted from the discriminative motion matrix with a higher accuracy compared to classical image-derived risk factors.Cette thèse présente de nouveaux outils informatiques pour quantifier les déformations et le mouvement de structures anatomiques à partir d’images médicales dans le cadre d’une grande variété d’applications cliniques. Des outils génériques de recalage déformable sont présentés qui permettent l’analyse de la déformation de tissus anatomiques pour améliorer le diagnostic, le pronostic et la thérapie. Ces outils combinent des méthodes avancées d’analyse d’images médicales avec des méthodes d’apprentissage automatique performantes.Dans un premier temps, nous nous concentrons sur les problèmes de recalages inter-sujets difficiles. En apprenant à partir d’exemples de déformation donnés, nous proposons un nouveau schéma d’optimisation basé sur un agent inspiré de l’apprentissage par renforcement profond dans lequel un modèle de déformation statistique est exploré de manière itérative montrant une précision améliorée de recalage. Dans un second temps, nous développons un modèle de déformation difféomorphe qui permet un recalage multi-échelle précis et une analyse de déformation en apprenant une représentation de faible dimension des déformations intra-sujet. La méthode non supervisée utilise un modèle de variable latente sous la forme d’un autoencodeur variationnel conditionnel (CVAE) pour apprendre une représentation probabiliste des déformations qui est utile pour la simulation, la classification et la comparaison des déformations. Troisièmement, nous proposons un modèle de mouvement probabiliste dérivé de séquences d’images d’organes en mouvement. Ce modèle génératif décrit le mouvement dans un espace latent structuré, la matrice de mouvement, qui permet le suivi cohérent des structures ainsi que l’analyse du mouvement. Ainsi cette approche permet la simulation et l’interpolation de modèles de mouvement réalistes conduisant à une acquisition et une augmentation des données plus rapides.Enfin, nous démontrons l’intérêt des outils développés dans une application clinique où le modèle de mouvement est utilisé pour le pronostic de maladies et la planification de thérapies. Il est démontré que le risque de survie des patients souffrant d’insuffisance cardiaque peut être prédit à partir de la matrice de mouvement discriminant avec une précision supérieure par rapport aux facteurs de risque classiques dérivés de l’image

Bayesian Deep Learning for Cardiac Motion Modelling and Analysis

Cardiovascular diseases (CVDs) remain a primary cause of mortality globally, with an estimated 17.9 million deaths in 2019, accounting for 32% of all global fatalities. In recent decades, non-invasive imaging, particularly Magnetic Resonance Imaging (MRI), has become pivotal in diagnosing CVDs, offering high-resolution, multidimensional, and sequential cardiac data. However, the interpretation of cardiac MRI data is challenging, due to the complexities of cardiac motion and anatomical variations. Traditional manual methods are time-consuming and subject to variability. Deep learning (DL) methods, notably generative models, have recently advanced medical image analysis, offering state-of-the-art solutions for segmentation, registration, and motion modelling. This thesis encapsulates the development and validation of deep-learning frameworks in the field of cardiac motion modelling and analysis from sequential cardiac MRI scans. At its core, it introduces a probabilistic generative model for cardiac motion modelling, underpinned by temporal coherence, capable of synthesising new CMR sequences. Three models are derived from this foundational probabilistic model, each contributing to different aspects. Firstly, through the innovative application of gradient surgery techniques, we address the dual objectives of attaining high registration accuracy and ensuring the diffeomorphic characteristics of the predicted motion fields. Subsequently, we introduce the joint operation of ventricular segmentation and motion modelling. The proposed method combines anatomical precision with the dynamic temporal flow to enhance both the accuracy of motion modelling and the stability of sequential segmentation. Furthermore, we introduce a conditional motion transfer framework that leverages variational models for the generation of cardiac motion, enabling anomaly detection and the augmentation of data, particularly for pathologies that are less commonly represented in datasets. This capability to transfer and transform cardiac motion across healthy and pathological domains is set to revolutionize how clinicians and researchers understand and interpret cardiac function and anomalies. Collectively, these advancements present novelty and application potentials in cardiac image processing. The methodologies proposed herein have the potential to transform routine clinical diagnostics and interventions, allowing for more nuanced and detailed cardiac assessments. The probabilistic nature of these models promises to deliver not only more detailed insights into cardiac health but also to foster the development of personalised medicine approaches in cardiology

Methods for three-dimensional Registration of Multimodal Abdominal Image Data

Multimodal image registration benefits the diagnosis, treatment planning and the performance of image-guided procedures in the liver, since it enables the fusion of complementary information provided by pre- and intrainterventional data about tumor localization and access. Although there exist various registration methods, approaches which are specifically optimized for the registration of multimodal abdominal scans are only scarcely available. The work presented in this thesis aims to tackle this problem by focusing on the development, optimization and evaluation of registration methods specifically for the registration of multimodal liver scans. The contributions to the research field of medical image registration include the development of a registration evaluation methodology that enables the comparison and optimization of linear and non-linear registration algorithms using a point-based accuracy measure. This methodology has been used to benchmark standard registration methods as well as novel approaches that were developed within the frame of this thesis. The results of the methodology showed that the employed similarity measure used during the registration has a major impact on the registration accuracy of the method. Due to this influence, two alternative similarity metrics bearing the potential to be used on multimodal image data are proposed and evaluated. The first metric relies on the use of gradient information in form of Histograms of Oriented Gradients (HOG) whereas the second metric employs a siamese neural network to learn a similarity measure directly on the image data. The evaluation showed, that both metrics could compete with state of the art similarity measures in terms of registration accuracy. The HOG-metric offers the advantage that it does not require ground truth data to learn a similarity estimation, but instead it is applicable to various data sets with the sole requirement of distinct gradients. However, the Siamese metric is characterized by a higher robustness for large rotations than the HOG-metric. To train such a network, registered ground truth data is required which may be critical for multimodal image data. Yet, the results show that it is possible to apply models trained on registered synthetic data on real patient data. The last part of this thesis focuses on methods to learn an entire registration process using neural networks, thereby offering the advantage to replace the traditional, time-consuming iterative registration procedure. Within the frame of this thesis, the so-called VoxelMorph network which was originally proposed for monomodal, non-linear registration learning is extended for affine and multimodal registration learning tasks. This extension includes the consideration of an image mask during metric evaluation as well as loss functions for multimodal data, such as the pretrained Siamese metric and a loss relying on the comparison of deformation fields. Based on the developed registration evaluation methodology, the performance of the original network as well as the extended variants are evaluated for monomodal and multimodal registration tasks using multiple data sets. With the extended network variants, it is possible to learn an entire multimodal registration process for the correction of large image displacements. As for the Siamese metric, the results imply a general transferability of models trained with synthetic data to registration tasks including real patient data. Due to the lack of multimodal ground truth data, this transfer represents an important step towards making Deep Learning based registration procedures clinically usable

Deep Learning in Cardiac Magnetic Resonance Image Analysis and Cardiovascular Disease Diagnosis

Cardiovascular diseases (CVDs) are the leading cause of death in the world, accounting for 17.9 million deaths each year, 31\% of all global deaths. According to the World Health Organisation (WHO), this number is expected to rise to 23 million by 2030. As a noninvasive technique, medical imaging with corresponding computer vision techniques is becoming more and more popular for detecting, understanding, and analysing CVDs. With the advent of deep learning, there are significant improvements in medical image analysis tasks (e.g. image registration, image segmentation, mesh reconstruction from image), achieving much faster and more accurate registration, segmentation, reconstruction, and disease diagnosis. This thesis focuses on cardiac magnetic resonance images, systematically studying critical tasks in CVD analysis, including image registration, image segmentation, cardiac mesh reconstruction, and CVD prediction/diagnosis. We first present a thorough review of deep learning-based image registration approaches, and subsequently, propose a novel solution to the problem of discontinuity-preserving intra-subject cardiac image registration, which is generally ignored in previous deep learning-based registration methods. On the basis of this, a joint segmentation and registration framework is further proposed to learn the joint relationship between these two tasks, leading to better registration and segmentation performance. In order to characterise the shape and motion of the heart in 3D, we present a deep learning-based 3D mesh reconstruction network that is able to recover accurate 3D cardiac shapes from 2D slice-wise segmentation masks/contours in a fast and robust manner. Finally, for CVD prediction/diagnosis, we design a multichannel variational autoencoder to learn the joint latent representation of the original cardiac image and mesh, resulting in a shape-aware image representation (SAIR) that serves as an explainable biomarker. SAIR has been shown to outperform traditional biomarkers in the prediction of acute myocardial infarction and the diagnosis of several other CVDs, and can supplement existing biomarkers to improve overall predictive performance

Leveraging Supervoxels for Medical Image Volume Segmentation With Limited Supervision

The majority of existing methods for machine learning-based medical image segmentation are supervised models that require large amounts of fully annotated images. These types of datasets are typically not available in the medical domain and are difficult and expensive to generate. A wide-spread use of machine learning based models for medical image segmentation therefore requires the development of data-efficient algorithms that only require limited supervision. To address these challenges, this thesis presents new machine learning methodology for unsupervised lung tumor segmentation and few-shot learning based organ segmentation. When working in the limited supervision paradigm, exploiting the available information in the data is key. The methodology developed in this thesis leverages automatically generated supervoxels in various ways to exploit the structural information in the images. The work on unsupervised tumor segmentation explores the opportunity of performing clustering on a population-level in order to provide the algorithm with as much information as possible. To facilitate this population-level across-patient clustering, supervoxel representations are exploited to reduce the number of samples, and thereby the computational cost. In the work on few-shot learning-based organ segmentation, supervoxels are used to generate pseudo-labels for self-supervised training. Further, to obtain a model that is robust to the typically large and inhomogeneous background class, a novel anomaly detection-inspired classifier is proposed to ease the modelling of the background. To encourage the resulting segmentation maps to respect edges defined in the input space, a supervoxel-informed feature refinement module is proposed to refine the embedded feature vectors during inference. Finally, to improve trustworthiness, an architecture-agnostic mechanism to estimate model uncertainty in few-shot segmentation is developed. Results demonstrate that supervoxels are versatile tools for leveraging structural information in medical data when training segmentation models with limited supervision

On Motion Analysis in Computer Vision with Deep Learning: Selected Case Studies

Motion analysis is one of the essential enabling technologies in computer vision. Despite recent significant advances, image-based motion analysis remains a very challenging problem. This challenge arises because the motion features are extracted directory from a sequence of images without any other meta data information. Extracting motion information (features) is inherently more difficult than in other computer vision disciplines. In a traditional approach, the motion analysis is often formulated as an optimisation problem, with the motion model being hand-crafted to reflect our understanding of the problem domain. The critical element of these traditional methods is a prior assumption about the model of motion believed to represent a specific problem. Data analytics’ recent trend is to replace hand-crafted prior assumptions with a model learned directly from observational data with no, or very limited, prior assumptions about that model. Although known for a long time, these approaches, based on machine learning, have been shown competitive only very recently due to advances in the so-called deep learning methodologies. This work's key aim has been to investigate novel approaches, utilising the deep learning methodologies, for motion analysis where the motion model is learned directly from observed data. These new approaches have focused on investigating the deep network architectures suitable for the effective extraction of spatiotemporal information. Due to the estimated motion parameters' volume and structure, it is frequently difficult or even impossible to obtain relevant ground truth data. Missing ground truth leads to choose the unsupervised learning methodologies which is usually represents challenging choice to utilize in already challenging high dimensional motion representation of the image sequence. The main challenge with unsupervised learning is to evaluate if the algorithm can learn the data model directly from the data only without any prior knowledge presented to the deep learning model during In this project, an emphasis has been put on the unsupervised learning approaches. Owning to a broad spectrum of computer vision problems and applications related to motion analysis, the research reported in the thesis has focused on three specific motion analysis challenges and corresponding practical case studies. These include motion detection and recognition, as well as 2D and 3D motion field estimation. Eyeblinks quantification has been used as a case study for the motion detection and recognition problem. The approach proposed for this problem consists of a novel network architecture processing weakly corresponded images in an action completion regime with learned spatiotemporal image features fused using cascaded recurrent networks. The stereo-vision disparity estimation task has been selected as a case study for the 2D motion field estimation problem. The proposed method directly estimates occlusion maps using novel convolutional neural network architecture that is trained with a custom-designed loss function in an unsupervised manner. The volumetric data registration task has been chosen as a case study for the 3D motion field estimation problem. The proposed solution is based on the 3D CNN, with a novel architecture featuring a Generative Adversarial Network used during training to improve network performance for unseen data. All the proposed networks demonstrated a state-of-the-art performance compared to other corresponding methods reported in the literature on a number of assessment metrics. In particular, the proposed architecture for 3D motion field estimation has shown to outperform the previously reported manual expert-guided registration methodology