Abbod, M

Amira, A

Mecheter, I

Zaidi, H

Brunel University Research Archive

Deep Learning based Segmentation for Multi MRImaging Protocols using Transfer Learning for PETAttenuation Correction1st Imene MecheterDepartment of Electronics and Computer EngineeringBrunel University LondonUxbridge, UKimene.mecheter@brunel.ac.uk2nd Abbes AmiraInstitute of Artificial IntelligenceDe Montfort UniversityLeicester, UK3rd Maysam AbbodDepartment of Electronics and Computer EngineeringBrunel University LondonUxbridge, UK4thHabib ZaidiDivision of Nuclear Medicine and Molecular ImagingGeneva University HospitalGeneva, SwitzerlandAbstract—Magnetic resonance (MR) image segmentation is arobust technique used for PET attenuation correction. However,the segmentation of the brain into different tissue classes is achallenging task because of the similarity of pixel intensity values.The objective of this work is to propose a deep learning networkto segment T1-weighted MR images of a dataset consists of 50patients. Additionally, transfer learning is applied to segmentanother MR image protocol which is T2-weighted. The pretrainednetwork with T1-weighted images is finetuned then tested with adataset of 14 patients only. The Dice coefficients of air, soft tissue,and bone classes for T1-weighted MR images are 0.98, 0.92 ,and 0.79 respectively. The results of transfer learning show thefeasibility of finetuning a deep network trained with T1-weightedimages to segment T2-weighted images.Index Terms—Magnetic Resonance Imaging, Segmentation,PET Attenuation Correction, Deep Learning, Transfer LearningI. INTRODUCTIONPositron emission tomography (PET) and magnetic reso-nance (MR) are two complemantry imaging modalities thatprovide physiological and morphological information of differ-ent tissues in both normal and pathological cases. Recently, thehybrid PET/MR imaging system has been commercialized andadopted in clinical domain [1]. The main challenge is the PETquantification process which requires the photon attenuatingproperties which cannot be derived directly from MR imageintensitties. The attenuation maps are commonly derived fromcomputed tomography (CT) images because of the directmapping between CT intensities and attenuation coefficients.Therefore, it is necessary to explore more complicated waysto convert the MR image intensities to 511 KeV attenuationmaps [2].Three methods have been performed to address the PETattenuation correction problem using MR imaging which are:segmentation, atlas, and emission methods [2]. The segmen-tation based method is considered the most robust and simplemethod which has been deployed in commercial scanners andintroduced in clinical practice [2]. Atlas based method showsits superiority, however, it is not considered yet as a robustmethod to be introduced in clinical domain [3].Deep learning networks have been recently applied onvarious computer vision applications. In the medical domain,different deep network architectures have been proposed formedical images segmentation [4] such as Fully convolutionalnetwork [5], Segnet [6], and U-Net [7].Deep networks are performing greatly with the assumptionthat the training and testing datasets follow the same datadistribution. However, this is very challenging with medicaldatasets due to the variation of commercial scanners and theavailability of different imaging modalities and protocols. Thisvariation is called domain shift [8]. Figure 1 shows the visualcomparison between T1-weighted (T1-w) MR protocol, T2-weighted (T2-w) MR protocol, and CT image where T1-wMR image looks darker than T2-w in most tissue classes.Fig. 1. Example of training datasets: T1-w and T2-w MR images with theircorresponding CT image.In this work, a deep convolutional auto encoder network isapplied to perform brain MR image segmentation into threeThis article has been accepted for publication in a future issue of this symposium, but has not been fully edited. Content may change prior to final publication. Citation information: DOI10.1109/SSCI47803.2020.9308177, 2020 IEEE Symposium Series on Computational Intelligence (SSCI)© 20XX IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.tissue classes. The network is trained from scratch using T1-wMR images. Afterwards, transfer learning is applied to finetunethe pretrained network to segment T2-w MR images.The structure of the paper is as follows: section 2 describesthe data. Section 3 presents the proposed methodology. Section4 illustrates the experimental results and evaluation. Finally theconclusion is covered in section 5.II. DATAA. Data acquisitionBrain MR and PET/CT images are acquired as part of theclinical workup of patients. The dataset consists of 50 patientsof T1-w MR images and 14 patients of T2-w MR images. Thepatients images show clinical indication of dementia (70%),epilepsy (25%) and brain tumors (5%). The age range of thepatients is 64.6 ± 11.7 years.At the first step, the patients underwent an MRI scan on a3T Siemens MAGNETOM Skyra scanner with a 64 channelhead coil. The MR images scans used for this study are 3DT1- w magnetization prepared rapid gradient-echo, MP-RAGE(TE/TR/TI, 2.3 ms/1900 ms/ 970 ms, flip angle 8; NEX =1, voxel size 0.8×0.8×0.8 mm3) and 3D T2-w turbo spin-echo, TSE (TE/TR, 100 ms/6200 ms, NEX = 2; voxel size0.4×0.4×4 mm3). Afterwards, the patients underwent an 18F-FDG PET/CT scan on the Siemens Biograph mCT scanner for20 min after injection of 210.2 ± 13.9 MBq 18F-FDG.B. Data preprocessingThe MR volumes of both T1-w and T2-w are converted into2D slices. The number of slices per each volume is differentand the dimension of each slice is 512 × 512. Each slice iscropped into a 256 × 256 image to remove the background.Then, local contrast normalization technique is applied on eachimage. The same processing steps are applied to CT volumeswhich are used as ground truth. The total number of 2D imagesfor the whole dataset is 2982 slices. Among them, only 2328slices are used after discarding some of the first and last slicesof each volume. The discarded slices have just few pixelsthat correspond to the brain tissue while the majority of thepixels correspond to the background which is not useful forthe training process.C. Ground truth generationCT images are used as ground truth for the supervisedtraining process. MR images are firstly co-registered with CTimages to determine a common coordinate system that enablesthe pixel-based comparison of images. Each MR slice is co-registered with its corresponding CT slice by applying therigid Euler transformation followed by the non-rigid B-splinetransformation using Elastix tool [9]. Afterward, simple pixelintensity-based thresholding is applied to create the CT groundtruth. The CT image is segmented into three tissue classeswhich are air, bone, and soft tissue. Hounsfield values whichare: greater than 600 HU are classified as bone, lower than-500 HU are classified as air, otherwise are labeled as softtissue.Fig. 2. An illustration of the SegNet architectureIII. PROPOSED METHODOLOGYThe proposed method consists of two parts. The first partis training a deep convolutional auto encoder (CAE) networkfrom scratch to segment T1-w MR images into three tissueclasses. The second part is finetuning the pre-trained networkwith T1-w images to segment T2-w MR images.A. Deep CAE Network to segment T1-w MRThe segmentation network architecture is based on theSegnet which follows the convolutional auto encoder architec-ture. SegNet is one common deep CAE model that has beenapplied successfully to segment the brain MR images [10],[11]. This model consists of an extracting path to capture thefeatures and an expanding path for localization and resizing.The output of such networks preserves the same size of theinput after extracting the meaningful features. The extractingpath consists of 13 convolutional layers with its correspondingbatch normalization layers, maxpooling layers, and rectified-linear unit (ReLU) activation function. Each convolutionallayer consists of multiple 3 × 3 filters. The expanding pathis the mirror of the extracting path with upsampling layers toretrain the original size of the input. The final layer of thenetwork is a multiclass softmax classifier that produces classprobabilities for each pixel. Figure 2 illustrates the networkarchitecture.The network is trained from scratch using T1-w MR datasetsusing two folds cross validation. A grid search strategy isfollowed to find the best training parameters. The networkweights are initialized using He normal scheme [12] andupdated using Adam optimizer with a learning rate that startswith 0.001 then reduces when there is no improvement on thetraining loss for more than 10 epochs. Beta 1 and Beta 2 whichare the parameters of Adam’s optimizer that controls the decayrates are set to 0.9 and 0.999 respectively. The network reads10 samples per each batch and calculates the multiclass cross-entropy loss for 300 epochs. Dropout with a factor of 0.7 isapplied to reduce the overfitting.B. Transfer learning to segment T2-w MRThe pretrained trained model with T1-w MR images isused as the base model to train a small dataset of T2-wMR images. The base model is finetuned by freezing theThis article has been accepted for publication in a future issue of this symposium, but has not been fully edited. Content may change prior to final publication. Citation information: DOI10.1109/SSCI47803.2020.9308177, 2020 IEEE Symposium Series on Computational Intelligence (SSCI)Fig. 3. An illustration of the transfer learning process to segment T2-w MRimagesextracting path while re-training the expanding path. Theprocess of freezing the parameters of specific layers preservesthe extracted features from T1-w MR images. The learningrate is set to a smaller value (1e-4) than the learning rate ofthe pretrained model to avoid destroying the learnt features.The number of epochs required to finetune the model is only30 epochs. Figure 3 illustrates the transfer learning process.C. Computing EnvironmentMATLAB and Python programming languages are usedto build the proposed method. The data preprocessing isperformed using MATLAB while Python libraries: Keras andTensorFlow are used for deep learning. The deep networkmodel is trained using Tesla V100 GPU with 16 GB RAMas part of ”Raad2” GPU cluster.IV. EXPERIMENTAL RESULTS AND EVALUATIONThe first conducted experiment is the training and testingof the Segnet network to segment T1-w MR images into threedifferent tissue classes: air, soft tissue, and bone. Then, thesecond experiment is the application of transfer learning byfinetuning the pretrained model with T1-w images to fit theT2-w images.A. Results of the segmentation of T1-w MR imagesThe segmentation results of T1-w MR images are illustratedin Figure 4. The first row corresponds to three input slices, thesecond row shows their corresponding CT images as groundtruth, and the last row represents the results of the segmen-tation. Table I shows the segmentation evaluation metrics oftesting dataset from T1-w MR testing images.The air and soft tissue classes achieve higher segmentationdice than the bone class. The challenging segmentation of bonetissue originates from the imbalance tissue classes as such thetotal number of air and soft tissue pixels per each slice is muchhigher than the bone pixels.Fig. 4. Example of three MR T1-w slices (A) with their corresponding CTslices as ground truth (B) and the automatic segmented slices using deeplearning(C). The colors in row B and C refer to the following classes: greenis the air class, yellow represents the soft tissue class, and the purple showsthe bone class.TABLE IEVALUATION METRICS OF THE SEGMENTATION RESULTS OF BRAIN T1-WMR IMAGESMetric Air Soft tissue BonePrecision 0.98 0.93 0.80Recall 0.98 0.93 0.77Dice 0.98 0.92 0.79Jaccard Index 0.96 0.86 0.65B. Results validationThe obtained segmentation results are validated with twoother approaches that applied deep learning to segment thebrain MR images for PET attenuation correction. Followingthe network architecture and training options adapted from[10], a segnet network is trained using our datasets. The secondapproach [13] applies the same segnet network architecturewith the addition of conditional random field as a post pro-cessing setp. The segmentation results of each approach areillustrated in table II.C. Results of transfer learning and finetuningThe results of the segmentation of T2-w MR images areshown in Figure 5. Table III illustrates the dice similarityTABLE IICOMPARISON WITH OTHER DEEP LEARNING BASED SEGMENTATIONMETHODSMethod Air Soft tissue BoneSegnet [10] 0.97 0.90 0.66Segnet+ CRF [13] 0.84 0.57 0.07Proposed method 0.98 0.92 0.79This article has been accepted for publication in a future issue of this symposium, but has not been fully edited. Content may change prior to final publication. Citation information: DOI10.1109/SSCI47803.2020.9308177, 2020 IEEE Symposium Series on Computational Intelligence (SSCI)Fig. 5. The segmentation results of T2-w MR images using transfer learning.Three MR T2-w slices are shown in row (A) with their corresponding CTslices as ground truth (B) and the automatic segmented slices using transferlearning(C). The colors in row B and C refer to the following classes: greenis the air class, yellow represents the soft tissue class, and the purple showsthe bone class.TABLE IIITHE DICE SIMILARITY COEFFICIENT OF THE T2-W MR BRAIN TISSUECLASSES: AIR, SOFT TISSUE, AND BONE USING TRANSFER LEARNING ANDFINETUNING THE T1-W PRETRAINED MODEL.Air Soft tissue BoneTransfer learning 0.97 0.85 0.51Finetune: 2 CONV 0.97 0.86 0.50Finetune: 4 CONV 0.97 0.87 0.58Finetune: 6 CONV 0.97 0.87 0.60Finetune: 8 CONV 0.97 0.87 0.60Finetune: Decoder 0.97 0.87 0.63coefficient of the three classes. The first row in the table refersto the results of transfer learning where the pretrained model isused as it is without any finetuning. The results show that thereis an improvement in the segmentation results as we finetunemore convolutional layers of the pretrained model. The bestresults are achieved by freezing the layers of the extractingpath (encoder) and retraining the layers of the expanding path(decoder).Like T1-w MR image segmentation, the air class achievesthe highest dice value followed by the the soft tissue class thenthe bone class. Despite the finetuning of more convolutionallayers helps to improve the segmentation accuracy of both thesoft tissue and bone class, the air class segmentation does notshow any improvement.The training time varies from training a network fromscratch to finetuning a pretrained model. The approximaterequired time of training T1-w MR images from scratch is6 hours. However, the finetuning of the pretrained modelrequires only 10 minutes. Moreover, the predication time isvery short such as labeling a single slice takes only 6 ms.V. CONCLUSIONIn this study, an MR based attenuation correction methodusing deep learning has been applied to segment the brainMR images into three tissue classes: air, soft tissue and bone.Transfer learning technique is applied to show the feasibility offinetuning a model trained with T1-w MR images to segmentT2-w MR images. The promising results indicate that there isa possibility to build a model that can be applied to segmentMR images with different MR protocols. The successfulapplication of deep learning can improve the segmentationaccuracy and hence improves the PET attenuation correctionand quantification.ACKNOWLEDGMENTThe authors extend a special thanks to Texas A&M atQatar’s Advanced Scientific Computing (TASC) center whoprovided the access and technical support to the High Perfor-mance Computing cluster ”Raad2”.REFERENCES[1] Z. Chen, S. D. Jamadar, S. Li, F. Sforazzini, J. Baran, N. Ferris, N. J.Shah, and G. F. Egan, “From simultaneous to synergistic mr-pet brainimaging: A review of hybrid mr-pet imaging methodologies,” Humanbrain mapping, vol. 39, no. 12, pp. 5126–5144, 2018.[2] A. Mehranian, H. Arabi, and H. Zaidi, “Vision 20/20: Magnetic reso-nance imaging-guided attenuation correction in PET/MRI: Challenges,solutions, and opportunities,” Medical Physics, vol. 43, no. 3, pp. 1130–1155, Mar. 2016.[3] K. Shi, S. Fürst, L. Sun, M. Lukas, N. Navab, S. Förster, and S. I. Ziegler,“Individual refinement of attenuation correction maps for hybrid pet/mrbased on multi-resolution regional learning,” Computerized MedicalImaging and Graphics, vol. 60, pp. 50–57, 2017.[4] Z. Akkus, A. Galimzianova, A. Hoogi, D. L. Rubin, and B. J. Erickson,“Deep learning for brain mri segmentation: state of the art and futuredirections,” Journal of digital imaging, vol. 30, no. 4, pp. 449–459, 2017.[5] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networksfor semantic segmentation,” in Proceedings of the IEEE conference oncomputer vision and pattern recognition, 2015, pp. 3431–3440.[6] V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep con-volutional encoder-decoder architecture for image segmentation,” IEEEtransactions on pattern analysis and machine intelligence, vol. 39,no. 12, pp. 2481–2495, 2017.[7] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networksfor biomedical image segmentation,” in International Conference onMedical image computing and computer-assisted intervention. Springer,2015, pp. 234–241.[8] T. Tommasi, M. Lanzi, P. Russo, and B. Caputo, “Learning the rootsof visual domain shift,” in European Conference on Computer Vision.Springer, 2016, pp. 475–482.[9] S. Klein, M. Staring, K. Murphy, M. A. Viergever, and J. P. Pluim,“Elastix: a toolbox for intensity-based medical image registration,” IEEEtransactions on medical imaging, vol. 29, no. 1, pp. 196–205, 2010.[10] F. Liu, H. Jang, R. Kijowski, T. Bradshaw, and A. B. McMillan, “Deeplearning mr imaging–based attenuation correction for pet/mr imaging,”Radiology, vol. 286, no. 2, pp. 676–684, 2017.[11] H. Jang, F. Liu, G. Zhao, T. Bradshaw, and A. B. McMillan, “TechnicalNote: Deep learning based MRAC using rapid ultrashort echo timeimaging,” Medical Physics, vol. 45, no. 8, pp. 3697–3704, 2018.[12] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers:Surpassing human-level performance on imagenet classification,” inProceedings of the IEEE international conference on computer vision,2015, pp. 1026–1034.[13] H. Jang, F. Liu, G. Zhao, T. Bradshaw, and A. B. McMillan, “Deeplearning based mrac using rapid ultrashort echo time imaging,” Medicalphysics, vol. 45, no. 8, pp. 3697–3704, 2018.This article has been accepted for publication in a future issue of this symposium, but has not been fully edited. Content may change prior to final publication. Citation information: DOI10.1109/SSCI47803.2020.9308177, 2020 IEEE Symposium Series on Computational Intelligence (SSCI)

Deep Learning based Segmentation for Multi MR Imaging Protocols using Transfer Learning for PET Attenuation Correction.

Crossref

Deep Learning based Segmentation for Multi MR Imaging Protocols using Transfer Learning for PET Attenuation Correction

http://bura.brunel.ac.uk/bitstream/2438/22251/3/FullText.pdf

Deep Learning based Segmentation for Multi MR Imaging Protocols using Transfer Learning for PET Attenuation Correction.

Abstract

Similar works

Full text

Available Versions

Brunel University Research Archive

Crossref