Label-driven weakly-supervised learning for multimodal deformable image registration

Spatially aligning medical images from different modalities remains a challenging task, especially for intraoperative applications that require fast and robust algorithms. We propose a weakly-supervised, label-driven formulation for learning 3D voxel correspondence from higher-level label correspondence, thereby bypassing classical intensity-based image similarity measures. During training, a convolutional neural network is optimised by outputting a dense displacement field (DDF) that warps a set of available anatomical labels from the moving image to match their corresponding counterparts in the fixed image. These label pairs, including solid organs, ducts, vessels, point landmarks and other ad hoc structures, are only required at training time and can be spatially aligned by minimising a cross-entropy function of the warped moving label and the fixed label. During inference, the trained network takes a new image pair to predict an optimal DDF, resulting in a fully-automatic, label-free, real-time and deformable registration. For interventional applications where large global transformation prevails, we also propose a neural network architecture to jointly optimise the global- and local displacements. Experiment results are presented based on cross-validating registrations of 111 pairs of T2-weighted magnetic resonance images and 3D transrectal ultrasound images from prostate cancer patients with a total of over 4000 anatomical labels, yielding a median target registration error of 4.2 mm on landmark centroids and a median Dice of 0.88 on prostate glands.Comment: Accepted to ISBI 201

A probabilistic deep learning model of inter-fraction anatomical variations in radiotherapy

In radiotherapy, the internal movement of organs between treatment sessions causes errors in the final radiation dose delivery. Motion models can be used to simulate motion patterns and assess anatomical robustness before delivery. Traditionally, such models are based on principal component analysis (PCA) and are either patient-specific (requiring several scans per patient) or population-based, applying the same deformations to all patients. We present a hybrid approach which, based on population data, allows to predict patient-specific inter-fraction variations for an individual patient. We propose a deep learning probabilistic framework that generates deformation vector fields (DVFs) warping a patient's planning computed tomography (CT) into possible patient-specific anatomies. This daily anatomy model (DAM) uses few random variables capturing groups of correlated movements. Given a new planning CT, DAM estimates the joint distribution over the variables, with each sample from the distribution corresponding to a different deformation. We train our model using dataset of 312 CT pairs from 38 prostate cancer patients. For 2 additional patients (22 CTs), we compute the contour overlap between real and generated images, and compare the sampled and ground truth distributions of volume and center of mass changes. With a DICE score of 0.86 and a distance between prostate contours of 1.09 mm, DAM matches and improves upon PCA-based models. The distribution overlap further indicates that DAM's sampled movements match the range and frequency of clinically observed daily changes on repeat CTs. Conditioned only on a planning CT and contours of a new patient without any pre-processing, DAM can accurately predict CTs seen during following treatment sessions, which can be used for anatomically robust treatment planning and robustness evaluation against inter-fraction anatomical changes

Prostate biopsy tracking with deformation estimation

Transrectal biopsies under 2D ultrasound (US) control are the current clinical standard for prostate cancer diagnosis. The isoechogenic nature of prostate carcinoma makes it necessary to sample the gland systematically, resulting in a low sensitivity. Also, it is difficult for the clinician to follow the sampling protocol accurately under 2D US control and the exact anatomical location of the biopsy cores is unknown after the intervention. Tracking systems for prostate biopsies make it possible to generate biopsy distribution maps for intra- and post-interventional quality control and 3D visualisation of histological results for diagnosis and treatment planning. They can also guide the clinician toward non-ultrasound targets. In this paper, a volume-swept 3D US based tracking system for fast and accurate estimation of prostate tissue motion is proposed. The entirely image-based system solves the patient motion problem with an a priori model of rectal probe kinematics. Prostate deformations are estimated with elastic registration to maximize accuracy. The system is robust with only 17 registration failures out of 786 (2%) biopsy volumes acquired from 47 patients during biopsy sessions. Accuracy was evaluated to 0.76$\pm$0.52mm using manually segmented fiducials on 687 registered volumes stemming from 40 patients. A clinical protocol for assisted biopsy acquisition was designed and implemented as a biopsy assistance system, which allows to overcome the draw-backs of the standard biopsy procedure.Comment: Medical Image Analysis (2011) epub ahead of prin

A comparative evaluation of 3 different free-form deformable image registration and contour propagation methods for head and neck MRI : the case of parotid changes radiotherapy

Purpose: To validate and compare the deformable image registration and parotid contour propagation process for head and neck magnetic resonance imaging in patients treated with radiotherapy using 3 different approachesthe commercial MIM, the open-source Elastix software, and an optimized version of it. Materials and Methods: Twelve patients with head and neck cancer previously treated with radiotherapy were considered. Deformable image registration and parotid contour propagation were evaluated by considering the magnetic resonance images acquired before and after the end of the treatment. Deformable image registration, based on free-form deformation method, and contour propagation available on MIM were compared to Elastix. Two different contour propagation approaches were implemented for Elastix software, a conventional one (DIR_Trx) and an optimized homemade version, based on mesh deformation (DIR_Mesh). The accuracy of these 3 approaches was estimated by comparing propagated to manual contours in terms of average symmetric distance, maximum symmetric distance, Dice similarity coefficient, sensitivity, and inclusiveness. Results: A good agreement was generally found between the manual contours and the propagated ones, without differences among the 3 methods; in few critical cases with complex deformations, DIR_Mesh proved to be more accurate, having the lowest values of average symmetric distance and maximum symmetric distance and the highest value of Dice similarity coefficient, although nonsignificant. The average propagation errors with respect to the reference contours are lower than the voxel diagonal (2 mm), and Dice similarity coefficient is around 0.8 for all 3 methods. Conclusion: The 3 free-form deformation approaches were not significantly different in terms of deformable image registration accuracy and can be safely adopted for the registration and parotid contour propagation during radiotherapy on magnetic resonance imaging. More optimized approaches (as DIR_Mesh) could be preferable for critical deformations

Bayesian modelling of organ deformations in radiotherapy

Moderne strålebehandling mot kreft er skreddarsydd for å gje ein høg stråledose tilpassa svulsten (målvolumet), mens så lite dose som mogleg vert gitt til det friske vevet omkring. Den totale dosen vert levert over nokre veker i daglege "fraksjonar", noko som reduserer biverknader. Under og mellom desse fraksjonane rører dei indre organa på seg heile tida på grunn av pust, fylling av blæra, tarmar si rørsle og ekstern påverknad. Likevel vert posisjonen til målvolumet og relevante risikoorgan bestemt på grunnlag av eit statisk 3D-skann som er tatt før behandlinga startar. Den vanlege måten å sikre seg mot konsekvensar av denne rørsla er å legge til marginar rundt svulsten. Slik sikrar ein å treffe målvolumet, men til gjengjeld får det friske vevet meir dose. Marginane sin storleik er fastsett ved hjelp av statistikk over tidlegare behandla pasienter. Dei statistiske metodane som vert brukte er ofte enkle, og tek berre omsyn til rigid rørsle, altså at heile kroppen rører seg i eitt. Dessutan vert det ikkje teke omsyn til rørsla til risikoorgan. For å berekne dose til risikoorgana er det vanleg å anta at forma til organa i planleggingsskannet er representative for forma deira under behandling. Arbeidet i denne avhandlinga handlar om å bruka teknikkar frå Bayesiansk statistikk for å modellere korleis organ rører og deformerer seg mellom fraksjonane. Målet er å estimere nøyaktig den statistiske fordelinga av rørsle for eit eller fleire organ til ein pasient. Fordelinga gjev innsikt i korleis organa forandre seg medan behandlinga går for seg. Denne innsikta er nyttig for evaluering av stråleterapiplanar, statistisk prediksjon av biverknader, såkalla robust planlegging og å berekna størrelsen på marginar. Metodane som vert presentert er evaluerte for endetarmen (rektum) sine rørsler hjå prostatakreftpasientar. For desse pasientane er rektum eit viktig risikoorgan, som kan bli ramma både av akutte og seine biverknader, som lekkasje, bløding og smerter. Samanlikna med eksisterande metodar har den Bayesianske tilnærminga to fordelar: For det første gir kombinasjonen av populasjonsstatistikk og individuelle data meir nøyaktige anslag av den pasientspesifikke fordelinga. For det andre estimerer dei nye metodane den såkalla systematiske feilen i tillegg til variasjonar frå fraksjon til fraksjon. Den systematiske feilen er forskjellen mellom den estimerte forma på organet under planlegging, og gjennomsnittsforma til organet under bestråling. Denne typen feil var tema for artikkel I. Her fekk vi til å redusere den systematiske feilen til rektum hjå 33 av 37 prostatakreftpasientar ved å bruke ein metode som kombinerer forma på rektum under planlegginga og gjennomsnittsforma i populasjonen. Vi vurderte og om denne forbetringa hadde påverknad på estimering av summert dose til rektum. Metoden gav ikkje signifikant forbetring for to antatt relevante parametrar (ekvivalent uniform dose og D5%), men gav signifikant reduksjon av bias på det estimerte dose-volum-histogrammet i intervallet 52.5 Gy til 65 Gy. Hovudarbeidet i dette prosjektet er publisert i artikkel II. Der presenterer vi to modellar for organrørsle basert på Bayesianske metodar. Inndata til desse metodane er organformer som er henta frå 3D-skanningar. Metodane kan ta ulikt tal slike former, og produserer meir nøyaktige resultat jo fleire former dei får. Dei gjev anslag av gjennomsnittsforma og kor stor uvissa om denne forma er, i tillegg til anslag av fordelinga av variasjon av former frå fraksjon til fraksjon. Vi evaluerte metodane etter kor godt dei kunne berekne "dekningssannsyn", altså sannsynet for at organet skal dekke eit gitt punkt i pasientkoordinatsystemet til ei gitt tid. For denne berekninga måtte titusenvis av organformer gjerast om til såkalla binærmasker, som er 3D-matriser av punkter i pasient-koordinatsystemet der verdien til eit punkt er 1 dersom punktet er inne i organet, og 0 elles. Denne berekninga var mogleg på grunn av programvare som blei implementert for dette prosjektet, og som er presentert i artikkel III. Også her var det prostatakreftpasientar sitt rektum som vart brukt til evaluering. Berekningane til dei nye metodane var likare det sanne dekningssannsynet enn tilsvarande berekningar frå tidlegare metodar, i signifikant grad, i alle fall opp til tre input. Forskjellen mellom dei to nye algoritmane er i hovudsak kompleksiteten og nøyaktigheita, og valet mellom algoritmane i ein gitt bruk vil vere ei avveging mellom desse faktorane. Vi viste ein måte modellane kan verte brukte i artikkel IV, som handlar om pasientar som får re-bestråling for tilbakefall av prostatakreft. Her brukte vi modellane til å berekne forventa akkumulert dose til rektum frå dei to behandlingane, og også uvissa rundt den forventa dosen. Metoden er basert på representative former" av rektum, altså former som rektum kan ta som er sannsynlege, men lite fordelaktige. Desse formene kan brukast som visuell hjelp for onkologar og doseplanleggjarar, og metoden kan implementerast ved hjelp av eksisterande funksjonar i programvaren for behandlingsplanlegging. Overordna gir denne avhandlinga nye løysingar for den sentrale utfordringa med å redusere konsekvensar av organrørsle i stråleterapi. Dei presenterte modellane er dei første som utnyttar statistikk for populasjonen og data frå den enkelte pasienten samstundes, og som tar omsyn til både systematiske og tilfeldige feil.Modern radiotherapy tends to be highly conformal, meaning that a high and uniform dose is delivered to the target volume and as little dose as possible to the surrounding normal tissue. The total radiation dose is delivered across several smaller daily fractions, typically spanning several weeks. During and between these fractions, internal organs are constantly in motion due to factors such as breathing, changes to bladder filling state, intestinal movement and external influences. Nevertheless, the position of the target and relevant organs at risk (OARs) are determined based on a static 3D scan acquired before start of treatment. A common safeguard which is used to take such motion into account is the addition of margins around the target. These margins reduce the chance of missing parts of the target, yet increases dose to the healthy tissue surrounding the target. The margin size is based on statistics from previous patients. However, for the most part, the statistical methods used are very simple, and typically based on an assumption of rigid patient motion. Similarly, motion of the OARs is commonly neglected. For estimation of dose to the OARs, it is common to assume that the organ shape at the static scan is representative for its shape during treatment. The work in this thesis concerns the use of techniques from Bayesian statistics for modelling inter-fraction organ motion and deformation. The goal is to estimate accurately the statistical distribution of shapes for one or more organs for a given patient. The distribution provides knowledge of how the patient's organs might move and deform during the radiotherapy course. This information is useful for the evaluation of radiotherapy plans, prediction of adverse effects, so-called motion-robust radiotherapy planning, the generation of margins and more. The methods presented in this thesis have been evaluated for predicting deformations of the rectum of prostate cancer patients. For these patients, the rectum is a crucial OAR that is affected by both early and late side effects including leakage, bleeding and pain. Compared to existing methods, the Bayesian approach developed and implemented in this thesis offers two advantages: first, combining population statistics and individual data leads to more accurate estimates of the patient-specific distribution. Secondly, the new methods estimate the distribution of the so-called systematic error in addition to variations from fraction to fraction. The systematic error is the difference between the estimated shape/position of an organ at the planning stage and its average shape/position during therapy, and was the subject of paper I. Here, we were able to reduce the systematic error of the rectum in 33 out of 37 prostate cancer patients using a straightforward method to combine the shape of the rectum at the planning CT with the population mean shape. We also evaluated the impact of this improvement on the estimation of dose to the rectum. We found no significant improvement on the estimation of two presumably relevant dose parameters (equivalent uniform dose and D5%). However, we did find significant reduction in the bias of the estimated dose-volume histogram in the range from 52.5 Gy to 65 Gy. Paper II contains the central work of this project. It presents two organ deformation models based on Bayesian methods. The input data to these algorithms are organ shapes derived from 3D scans. The methods can take a varying number of such inputs from a given patient, and will produce more accurate results the more inputs they are given. They provide an estimate of the mean shape of the organ, as well as the uncertainty of this mean, in addition to the distribution of the variation of shapes from fraction to fraction. The methods were evaluated in the task of estimating coverage probabilities, i.e. the probability that the organ will cover a certain point in the patient coordinate system, for the rectum of prostate cancer patients. For this evaluation, tens of thousands of organ shapes needed to be converted to so-called binary masks, which are 3D arrays of points in the patient coordinate system where the value of each point is 1 if the point is inside the organ and 0 if it is outside. This was enabled by the highly efficient point-in-polyhedron software presented in paper III, which was developed for this project. The models were given varying number of scans, from 1 to 10, as input, and compared to two existing (non-Bayesian) models. The estimates of the coverage probability produced by the new models were significantly more similar to the ground truth than those produced by the existing models, at least up to three input scans. The main differences between the two new algorithms are their of conceptual complexity and accuracy, and the choice of method in a given application will therefore come down to a trade-off between these qualities. An application for the models derived in paper II, concerning patients receiving re-irradiation for recurrent prostate cancer, is presented in paper IV. We introduce a way of estimating the expectation and uncertainty of the accumulated dose to the rectum from the two treatment courses. The method is based on "representative shapes" of the rectum, that is, shapes that are probable and also particularly favourable or unfavourable in terms of dose. The advantage is that these shapes can be used as a visual aid for the oncologist or dose planner, and that the method can be implemented using existing features of treatment planning systems. Overall, this thesis provides novel solutions to the central challenge of organ motion mitigation in RT. The presented models are the first to simultaneously exploit population and patient specific organ motion and addressing both systematic and random errors.Doktorgradsavhandlin

Learning statistical correlation for fast prostate registration in image-guided radiotherapy: Learning statistical correlation for radiotherapy

Purpose: In adaptive radiation therapy of prostate cancer, fast and accurate registration between the planning image and treatment images of the patient is of essential importance. With the authors’ recently developed deformable surface model, prostate boundaries in each treatment image can be rapidly segmented and their correspondences (or relative deformations) to the prostate boundaries in the planning image are also established automatically. However, the dense correspondences on the nonboundary regions, which are important especially for transforming the treatment plan designed in the planning image space to each treatment image space, are remained unresolved. This paper presents a novel approach to learn the statistical correlation between deformations of prostate boundary and nonboundary regions, for rapidly estimating deformations of the nonboundary regions when given the deformations of the prostate boundary at a new treatment image

Analysis of contrast-enhanced medical images.

Early detection of human organ diseases is of great importance for the accurate diagnosis and institution of appropriate therapies. This can potentially prevent progression to end-stage disease by detecting precursors that evaluate organ functionality. In addition, it also assists the clinicians for therapy evaluation, tracking diseases progression, and surgery operations. Advances in functional and contrast-enhanced (CE) medical images enabled accurate noninvasive evaluation of organ functionality due to their ability to provide superior anatomical and functional information about the tissue-of-interest. The main objective of this dissertation is to develop a computer-aided diagnostic (CAD) system for analyzing complex data from CE magnetic resonance imaging (MRI). The developed CAD system has been tested in three case studies: (i) early detection of acute renal transplant rejection, (ii) evaluation of myocardial perfusion in patients with ischemic heart disease after heart attack; and (iii), early detection of prostate cancer. However, developing a noninvasive CAD system for the analysis of CE medical images is subject to multiple challenges, including, but are not limited to, image noise and inhomogeneity, nonlinear signal intensity changes of the images over the time course of data acquisition, appearances and shape changes (deformations) of the organ-of-interest during data acquisition, determination of the best features (indexes) that describe the perfusion of a contrast agent (CA) into the tissue. To address these challenges, this dissertation focuses on building new mathematical models and learning techniques that facilitate accurate analysis of CAs perfusion in living organs and include: (i) accurate mathematical models for the segmentation of the object-of-interest, which integrate object shape and appearance features in terms of pixel/voxel-wise image intensities and their spatial interactions; (ii) motion correction techniques that combine both global and local models, which exploit geometric features, rather than image intensities to avoid problems associated with nonlinear intensity variations of the CE images; (iii) fusion of multiple features using the genetic algorithm. The proposed techniques have been integrated into CAD systems that have been tested in, but not limited to, three clinical studies. First, a noninvasive CAD system is proposed for the early and accurate diagnosis of acute renal transplant rejection using dynamic contrast-enhanced MRI (DCE-MRI). Acute rejection–the immunological response of the human immune system to a foreign kidney–is the most sever cause of renal dysfunction among other diagnostic possibilities, including acute tubular necrosis and immune drug toxicity. In the U.S., approximately 17,736 renal transplants are performed annually, and given the limited number of donors, transplanted kidney salvage is an important medical concern. Thus far, biopsy remains the gold standard for the assessment of renal transplant dysfunction, but only as the last resort because of its invasive nature, high cost, and potential morbidity rates. The diagnostic results of the proposed CAD system, based on the analysis of 50 independent in-vivo cases were 96% with a 95% confidence interval. These results clearly demonstrate the promise of the proposed image-based diagnostic CAD system as a supplement to the current technologies, such as nuclear imaging and ultrasonography, to determine the type of kidney dysfunction. Second, a comprehensive CAD system is developed for the characterization of myocardial perfusion and clinical status in heart failure and novel myoregeneration therapy using cardiac first-pass MRI (FP-MRI). Heart failure is considered the most important cause of morbidity and mortality in cardiovascular disease, which affects approximately 6 million U.S. patients annually. Ischemic heart disease is considered the most common underlying cause of heart failure. Therefore, the detection of the heart failure in its earliest forms is essential to prevent its relentless progression to premature death. While current medical studies focus on detecting pathological tissue and assessing contractile function of the diseased heart, this dissertation address the key issue of the effects of the myoregeneration therapy on the associated blood nutrient supply. Quantitative and qualitative assessment in a cohort of 24 perfusion data sets demonstrated the ability of the proposed framework to reveal regional perfusion improvements with therapy, and transmural perfusion differences across the myocardial wall; thus, it can aid in follow-up on treatment for patients undergoing the myoregeneration therapy. Finally, an image-based CAD system for early detection of prostate cancer using DCE-MRI is introduced. Prostate cancer is the most frequently diagnosed malignancy among men and remains the second leading cause of cancer-related death in the USA with more than 238,000 new cases and a mortality rate of about 30,000 in 2013. Therefore, early diagnosis of prostate cancer can improve the effectiveness of treatment and increase the patient’s chance of survival. Currently, needle biopsy is the gold standard for the diagnosis of prostate cancer. However, it is an invasive procedure with high costs and potential morbidity rates. Additionally, it has a higher possibility of producing false positive diagnosis due to relatively small needle biopsy samples. Application of the proposed CAD yield promising results in a cohort of 30 patients that would, in the near future, represent a supplement of the current technologies to determine prostate cancer type. The developed techniques have been compared to the state-of-the-art methods and demonstrated higher accuracy as shown in this dissertation. The proposed models (higher-order spatial interaction models, shape models, motion correction models, and perfusion analysis models) can be used in many of today’s CAD applications for early detection of a variety of diseases and medical conditions, and are expected to notably amplify the accuracy of CAD decisions based on the automated analysis of CE images