In this work, we propose a method to aid the 2-D segmentation of short-axis cardiac MRI. In particular, the deformation fields obtained during the registration are used to propagate the labels to all time frames, resulting in a weakly supervised segmentation approach that benefits from the features in unlabelled volumes along with the annotated data. Experimental results over the M\&Ms datasets show that the addition of the synthetically obtained labels to the original dataset yields promising results in the performance and improves the capability of the network to generalise to scanners from different vendors

Curran, Kathleen M.

Garcia-Cabrera, Carles

McGuinness, Kevin

O'Connor, Noel E.

DCU Online Research Access Service

Semi-supervised Learning of Cardiac MRI using Image RegistrationCarles Garcia-Cabrera1, Kathleen M. Curran2, Noel E. O’Connor1, and Kevin McGuinness11Dublin City University2University College DublinAbstractIn this work, we propose a method to aid the 2-D segmentation of short-axis cardiac MRI. In particular, thedeformation fields obtained during the registration are used to propagate the labels to all time frames, resultingin a weakly supervised segmentation approach that benefits from the features in unlabelled volumes along with theannotated data. Experimental results over the M&Ms datasets show that the addition of the synthetically obtainedlabels to the original dataset yields promising results in the performance and improves the capability of the networkto generalise to scanners from different vendors.Keywords: Cardiac MRI, Image Segmentation, Semi-Supervised Learning, Image Registration, Medical Imaging.1 IntroductionCardiac image segmentation is an important first step for many approaches to quantitative analysis for cardiac diagnosticassessment. This process requires partitioning the image into a number of clinically meaningful regions such as leftventricle (LV), right ventricle (RV), or myocardium (MYO). Acquiring this information allows clinicians understandimportant features such as the ejection fraction and the volume that the heart is managing at different times. Thosefeatures are later used to determine if there is any possible pathology and how bad it is [Kawel-Boehm et al., 2015].Data is a key challenge when trying to use off-the-shelf algorithms in this area, specifically the limited amount ofannotated data available, but also its quality. Many researchers report struggling to achieve improved results with existingannotated data, especially when working with open datasets [Chen et al., 2020]. Difficulties range from low availabilityof data, to domain shift using data from a certain scanner vendors, to images from patients with rare conditions.There is growing interest in the community in understanding how to transfer models that work well for specificscanners to unseen ones. In this work we address these challenges using the M&Ms challenge dataset, which webelieve to be the representative dataset for this issue [Campello et al., 2020].Convolutional neural networks (CNN) are the most common type of deep neural networks for image analysis andhave advanced the state-of-the-art in many object segmentation tasks, including in the medical imaging domain. Inparticular, U-Net [Ronneberger et al., 2015] is the architecture with the best results over most of the challenges thatthe cardiac MRI currently faces [Chen et al., 2020]. Hence, we have selected it to run our experiments.Recently, [Zhang et al., 2021] proposed a method where labels can be propagated using image registration in anunsupervised manner and those labels are used to enhance the process where a 3D U-Net learns features. Even thoughresults ranked second in the M&Ms challenge [Campello et al., 2020], the study did not test those propositions in 2Dnetworks, a key novelty of our work, nor explored different tools for the registration part.Contributions: this work proposes a method to enhance the segmentation of short-axis cardiac MRI by syntheticallylabelling volumes without annotations. The proposed system works by registering the labelled volumes of the end-systolic (ES) and the end-diastolic (ED) time-frames and using the warping field over their ground truth. This helps thenetwork to learn robust features from those volumes, which are important towards increasing the performance over differ-ent scanners. Our study extends recent works where the propagation of labels is used in a 3D U-Net and using a differentregistration tool. In contrast to that work, we investigate whether this technique is valid for 2D U-Net applications thatcan be used in less computationally powerful machines and expect to have better performances in anisotropic datasets.2 Related WorkThe clinical interest in ventricle segmentation has pushed the community towards improving the performance for thistask. With the advent of deep learning, a succession of different approaches were investigated. One of the first, was theusage of a fully convolutional network [Tran, 2016]. From there, many works have improved the networks, increasingthe learning capacity for segmentation [Khened et al., 2019]. In that regard, the low through-plane resolution and themotion artifacts limited the applicability of 3D networks [Baumgartner et al., 2017].Annotated data is the foundation of fully supervised approaches but at the same time it is scarce and costly toobtain. Moreover, annotating medical images requires significant expertise and manual effort and even then can lead tonoisy labels and/or biases. Additionally, there is an imbalance between the amount of available data regarding differentsource scanner, target pathology, or scanning parameters. All the above-mentioned challenges point strongly towardsthe need to design methods where the accuracy of the networks is maintained compared to fully supervised approach,even when there is significantly less data available.A number of works in the literature proposed unsupervised or semi-supervised techniques to overcome the scarcityof labels. One approach was to use a scribble annotation, that consists of a set of quickly drawn labels, and recursivelyre-training the network using the output segmentation, including a conditional random field and an uncertainty estimator[Can et al., 2018]. The advances in the calculation of optical flow inspired the usage of these algorithms for motionestimation and this has been useful in multi-task approaches where a Siamese network performs the motion estimationand the segmentation simultaneously, exploiting the information contained in unlabelled data [Qin et al., 2018].Voxelmorph [Balakrishnan et al., 2018] is an image registration tool for alignment and registration that can alsomodel deformations. It has been used for atlas based registration [Dalca et al., 2019] and for probabilistic diffeomorphicregistration [Dalca et al., 2018]. Given its proven reliability, we use it for the registration part of our study.3 Method and ExperimentThis section describes the proposed method, detailing the different steps and implementations of all the tools involvedin the process. Distinguishing two first steps corresponding to the registration and label propagation, and two last partscorresponding to training and testing the two different models i.e. the original dataset and the one with the additionof the synthetic labels.3.1 DataFor our experiments we used the M&Ms dataset, released during the Multi-Centre, Multi-Vendor and Multi-DiseaseCardiac Image Segmentation Challenge. This dataset consists of 345 patients with hypertrophic and dilated cardiomy-opathies and healthy subjects and it also permits the opportunity to measure performance over different scanners, andto evaluate the resilience of our approach towards different data sources. In particular the training set includes 150cases (75 from vendor A and 75 from vendor B). The test set includes cases from two more vendors (C and D).The CMR images have been segmented by experienced clinicians from the respective host institutions, includingcontours for the LV and RV blood pools, as well as for the left ventricular MYO.3.2 Label PropagationOur selected image registration technique was Voxelmorph [Balakrishnan et al., 2019], and given that the pre-trainedmodels were trained with images from other tissues, we decided to train our own with our available data. We alsoincluded the ground truth segmentation labels in the training process to produce a model that later we used for labelpropagation. For this part of the process we used the standard configuration of Voxelmorph (Tensorflow version)1.After training the model we computed all the warping fields for the time-frame between ED and ES, always in anintra-subject way. With the warping fields we modified the ground truth for the ED time-frame, obtaining the syntheticlabels for out target time-frames.Lastly, we join in a single dataset all the volumes with annotations, including the the original and the synthetic.3.3 SegmentationIn our approach, we used a 2D U-Net to segment the end-diastolic and end-systolic volumes in the target dataset.In particular, our U-Net model used 32 feature channels with kernel size equal to three in the first level of theconvolutional layers, where batch normalization was also applied and the activation function was ReLU, followedby a 2×2 max-pooling layer. The final 1×1 convolution is set with four channels that match the four target regions:background, right ventricle, left ventricle, and myocardium.To train the model, we used an Adam optimizer (learning rate = 10−3) with a plateau learning rate scheduler, andas a loss function we calculated the cross-entropy loss. In addition to this, we computed the Dice score for all theregions in each epoch for the validation set.4 ResultsTable 1 shows the results obtained for both datasets, including regions and vendors (M&Ms). The proposed methodperforms 2.6% better than the baseline, where the region that benefits the most from this change is the correspondingto the myocardium. When we compare the difference in the performance of both models in the different scanners,we see how the performance of unseen scanners C and D improves in the model resulting from the proposed methodwhile it sees its performance almost untouched for the scanners present in the training set (A and B).Regions Vendors TotalLV Myo RV A B C D DiceBaseline (M&Ms) 0.475 0.386 0.394 0.553 0.583 0.419 0.277 0.418Proposed (M&Ms) 0.504 0.430 0.398 0.548 0.573 0.443 0.338 0.444Table 1: Results (dice scores) on the M&Ms datasets. Higher is better.5 ConclusionsExperimental results using synthetic labels generated from VoxelMorph showed an improvement over the base-line, demonstrating the potential for using diffeomorphic image registration as a label propagation technique. Fu-ture work will investigate using this technique with a stronger baseline. In particular, we plan to adopt nnUNet[Isensee et al., 2021], an automatic segmentation framework for medical images based on the U-net architectures,which has been demonstrated to give state-of-the-art results. Finally, we believe the results can be improved byleveraging research on unreliable or noisy labels (e.g. [Arazo et al., 2019]), weighting them differently to reflect thelevel of trust in the annotations, or post-processing the synthetic annotations. Future work will investigate this.AcknowledgmentsThis work was funded by Science Foundation Ireland through the SFI Centre for Research Training in MachineLearning (18/CRT/6183).1VoxelMorph https://github.com/voxelmorph/voxelmorph.References[Arazo et al., 2019] Arazo, E., Ortego, D., Albert, P., O’Connor, N., and McGuinness, K. (2019). Unsupervised labelnoise modeling and loss correction. In International Conference on Machine Learning, pages 312–321. PMLR.[Balakrishnan et al., 2018] Balakrishnan, G., Zhao, A., Sabuncu, M. R., Dalca, A. V., and Guttag, J. (2018). Anunsupervised learning model for deformable medical image registration. Computer Vision and Pattern Recognition(CVPR).[Balakrishnan et al., 2019] Balakrishnan, G., Zhao, A., Sabuncu, M. R., Guttag, J., and Dalca, A. V. (2019).Voxelmorph: A learning framework for deformable medical image registration. IEEE Transactions on MedicalImaging, 38(8):1788–1800.[Baumgartner et al., 2017] Baumgartner, C. F., Koch, L. M., Pollefeys, M., and Konukoglu, E. (2017). An explorationof 2d and 3d deep learning techniques for cardiac mr image segmentation.[Campello et al., 2020] Campello, V. M., Palomares, J. F. R., Guala, A., Marakas, M., Friedrich, M., and Lekadir,K. (2020). Multi-Centre, Multi-Vendor & Multi-Disease Cardiac Image Segmentation Challenge.[Can et al., 2018] Can, Y. B., Chaitanya, K., Mustafa, B., Koch, L. M., Konukoglu, E., and Baumgartner, C. F. (2018).Learning to segment medical images with scribble-supervision alone.[Chen et al., 2020] Chen, C., Qin, C., Qiu, H., Tarroni, G., Duan, J., Bai, W., and Rueckert, D. (2020). Deep learningfor cardiac image segmentation: A review. Frontiers in Cardiovascular Medicine, 7.[Dalca et al., 2018] Dalca, A. V., Balakrishnan, G., Guttag, J., and Sabuncu, M. R. (2018). Unsupervised learningfor fast probabilistic diffeomorphic registration. Lecture Notes in Computer Science, page 729–738.[Dalca et al., 2019] Dalca, A. V., Rakic, M., Guttag, J., and Sabuncu, M. R. (2019). Learning conditional deformabletemplates with convolutional networks.[Isensee et al., 2021] Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J., and Maier-Hein, K. H. (2021). nnU-Net: aself-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 18(2):203–211.[Kawel-Boehm et al., 2015] Kawel-Boehm, N., Maceira, A., Valsangiacomo-Buechel, E. R., Vogel-Claussen,J., Turkbey, E. B., Williams, R., Plein, S., Tee, M., Eng, J., and Bluemke, D. A. (2015). Normal values forcardiovascular magnetic resonance in adults and children. Journal of Cardiovascular Magnetic Resonance, 17(1):29.[Khened et al., 2019] Khened, M., Kollerathu, V. A., and Krishnamurthi, G. (2019). Fully convolutional multi-scaleresidual densenets for cardiac segmentation and automated cardiac diagnosis using ensemble of classifiers. MedicalImage Analysis, 51:21 – 45.[Qin et al., 2018] Qin, C., Bai, W., Schlemper, J., Petersen, S. E., Piechnik, S. K., Neubauer, S., and Rueckert, D.(2018). Joint learning of motion estimation and segmentation for cardiac mr image sequences.[Ronneberger et al., 2015] Ronneberger, O., Fischer, P., and Brox, T. (2015). U-Net: Convolutional networks forbiomedical image segmentation.[Tran, 2016] Tran, P. V. (2016). A fully convolutional neural network for cardiac segmentation in short-axis MRI.[Zhang et al., 2021] Zhang, Y., Yang, J., Hou, F., Liu, Y., Wang, Y., Tian, J., Zhong, C., Zhang, Y., and He, Z. (2021).Semi-supervised cardiac image segmentation via label propagation and style transfer. In Statistical Atlases andComputational Models of the Heart. M&Ms and EMIDEC Challenges, pages 219–227. Springer.

Semi-supervised learning of cardiac MRI using image registration

http://doras.dcu.ie/26161/1/IMVIP_2021_LabelPropCardiac.pdf

Semi-supervised learning of cardiac MRI using image registration

Abstract

Similar works

Full text

Available Versions

DCU Online Research Access Service