One-shot Localization and Segmentation of Medical Images with Foundation Models

Recent advances in Vision Transformers (ViT) and Stable Diffusion (SD) models with their ability to capture rich semantic features of the image have been used for image correspondence tasks on natural images. In this paper, we examine the ability of a variety of pre-trained ViT (DINO, DINOv2, SAM, CLIP) and SD models, trained exclusively on natural images, for solving the correspondence problems on medical images. While many works have made a case for in-domain training, we show that the models trained on natural images can offer good performance on medical images across different modalities (CT,MR,Ultrasound) sourced from various manufacturers, over multiple anatomical regions (brain, thorax, abdomen, extremities), and on wide variety of tasks. Further, we leverage the correspondence with respect to a template image to prompt a Segment Anything (SAM) model to arrive at single shot segmentation, achieving dice range of 62%-90% across tasks, using just one image as reference. We also show that our single-shot method outperforms the recently proposed few-shot segmentation method - UniverSeg (Dice range 47%-80%) on most of the semantic segmentation tasks(six out of seven) across medical imaging modalities.Comment: Accepted at NeurIPS 2023 R0-FoMo Worksho

Region of Interest focused MRI to Synthetic CT Translation using Regression and Classification Multi-task Network

In this work, we present a method for synthetic CT (sCT) generation from zero-echo-time (ZTE) MRI aimed at structural and quantitative accuracies of the image, with a particular focus on the accurate bone density value prediction. We propose a loss function that favors a spatially sparse region in the image. We harness the ability of a multi-task network to produce correlated outputs as a framework to enable localisation of region of interest (RoI) via classification, emphasize regression of values within RoI and still retain the overall accuracy via global regression. The network is optimized by a composite loss function that combines a dedicated loss from each task. We demonstrate how the multi-task network with RoI focused loss offers an advantage over other configurations of the network to achieve higher accuracy of performance. This is relevant to sCT where failure to accurately estimate high Hounsfield Unit values of bone could lead to impaired accuracy in clinical applications. We compare the dose calculation maps from the proposed sCT and the real CT in a radiation therapy treatment planning setup

Evaluation of Sinus/Edge-Corrected Zero-Echo-Time–Based Attenuation Correction in Brain PET/MRI

In brain PET/MRI, the major challenge of zero-echo-time (ZTE)-based attenuation correction (ZTAC) is the misclassification of air/tissue/bone mixtures or their boundaries. Our study aimed to evaluate a sinus/edge-corrected (SEC) ZTAC (ZTACSEC), relative to an uncorrected (UC) ZTAC (ZTACUC) and a CT atlas-based attenuation correction (ATAC). Methods: Whole-body 18F-FDG PET/MRI scans were obtained for 12 patients after PET/CT scans. Only data acquired at a bed station that included the head were used for this study. Using PET data from PET/MRI, we applied ZTACUC, ZTACSEC, ATAC, and reference CT-based attenuation correction (CTAC) to PET attenuation correction. For ZTACUC, the bias-corrected and normalized ZTE was converted to pseudo-CT with air (-1,000 HU for ZTE < 0.2), soft-tissue (42 HU for ZTE > 0.75), and bone (-2,000 × [ZTE - 1] + 42 HU for 0.2 ≤ ZTE ≤ 0.75). Afterward, in the pseudo-CT, sinus/edges were automatically estimated as a binary mask through morphologic processing and edge detection. In the binary mask, the overestimated values were rescaled below 42 HU for ZTACSEC For ATAC, the atlas deformed to MR in-phase was segmented to air, inner air, soft tissue, and continuous bone. For the quantitative evaluation, PET mean uptake values were measured in twenty 1-mL volumes of interest distributed throughout brain tissues. The PET uptake was compared using a paired t test. An error histogram was used to show the distribution of voxel-based PET uptake differences. Results: Compared with CTAC, ZTACSEC achieved the overall PET quantification accuracy (0.2% ± 2.4%, P = 0.23) similar to CTAC, in comparison with ZTACUC (5.6% ± 3.5%, P < 0.01) and ATAC (-0.9% ± 5.0%, P = 0.03). Specifically, a substantial improvement with ZTACSEC (0.6% ± 2.7%, P < 0.01) was found in the cerebellum, in comparison with ZTACUC (8.1% ± 3.5%, P < 0.01) and ATAC (-4.1% ± 4.3%, P < 0.01). The histogram of voxel-based uptake differences demonstrated that ZTACSEC reduced the magnitude and variation of errors substantially, compared with ZTACUC and ATAC. Conclusion: ZTACSEC can provide an accurate PET quantification in brain PET/MRI, comparable to the accuracy achieved by CTAC, particularly in the cerebellum

Clinical evaluation of zero-echo-time MR imaging for the segmentation of the skull

MR-based attenuation correction is instrumental for integrated PET/MR imaging. It is generally achieved by segmenting MR images into a set of tissue classes with known attenuation properties (e.g., air, lung, bone, fat, soft tissue). Bone identification with MR imaging is, however, quite challenging, because of the low proton density and fast decay time of bone tissue. The clinical evaluation of a novel, recently published method for zero-echo-time (ZTE)-based MR bone depiction and segmentation in the head is presented here. METHODS: A new paradigm for MR imaging bone segmentation, based on proton density-weighted ZTE imaging, was disclosed earlier in 2014. In this study, we reviewed the bone maps obtained with this method on 15 clinical datasets acquired with a PET/CT/MR trimodality setup. The CT scans acquired for PET attenuation-correction purposes were used as reference for the evaluation. Quantitative measurements based on the Jaccard distance between ZTE and CT bone masks and qualitative scoring of anatomic accuracy by an experienced radiologist and nuclear medicine physician were performed. RESULTS: The average Jaccard distance between ZTE and CT bone masks evaluated over the entire head was 52% ± 6% (range, 38%-63%). When only the cranium was considered, the distance was 39% ± 4% (range, 32%-49%). These results surpass previously reported attempts with dual-echo ultrashort echo time, for which the Jaccard distance was in the 47%-79% range (parietal and nasal regions, respectively). Anatomically, the calvaria is consistently well segmented, with frequent but isolated voxel misclassifications. Air cavity walls and bone/fluid interfaces with high anatomic detail, such as the inner ear, remain a challenge. CONCLUSION: This is the first, to our knowledge, clinical evaluation of skull bone identification based on a ZTE sequence. The results suggest that proton density-weighted ZTE imaging is an efficient means of obtaining high-resolution maps of bone tissue with sufficient anatomic accuracy for, for example, PET attenuation correction

Zero-Echo-Time and Dixon Deep Pseudo-CT (ZeDD CT): Direct Generation of Pseudo-CT Images for Pelvic PET/MRI Attenuation Correction Using Deep Convolutional Neural Networks with Multiparametric MRI

Accurate quantification of uptake on PET images depends on accurate attenuation correction in reconstruction. Current MR-based attenuation correction methods for body PET use a fat and water map derived from a 2-echo Dixon MRI sequence in which bone is neglected. Ultrashort-echo-time or zero-echo-time (ZTE) pulse sequences can capture bone information. We propose the use of patient-specific multiparametric MRI consisting of Dixon MRI and proton-density-weighted ZTE MRI to directly synthesize pseudo-CT images with a deep learning model: we call this method ZTE and Dixon deep pseudo-CT (ZeDD CT). Methods: Twenty-six patients were scanned using an integrated 3-T time-of-flight PET/MRI system. Helical CT images of the patients were acquired separately. A deep convolutional neural network was trained to transform ZTE and Dixon MR images into pseudo-CT images. Ten patients were used for model training, and 16 patients were used for evaluation. Bone and soft-tissue lesions were identified, and the SUVmax was measured. The root-mean-squared error (RMSE) was used to compare the MR-based attenuation correction with the ground-truth CT attenuation correction. Results: In total, 30 bone lesions and 60 soft-tissue lesions were evaluated. The RMSE in PET quantification was reduced by a factor of 4 for bone lesions (10.24% for Dixon PET and 2.68% for ZeDD PET) and by a factor of 1.5 for soft-tissue lesions (6.24% for Dixon PET and 4.07% for ZeDD PET). Conclusion: ZeDD CT produces natural-looking and quantitatively accurate pseudo-CT images and reduces error in pelvic PET/MRI attenuation correction compared with standard methods

Zero-Echo-Time and Dixon Deep Pseudo-CT (ZeDD CT): Direct Generation of Pseudo-CT Images for Pelvic PET/MRI Attenuation Correction Using Deep Convolutional Neural Networks with Multiparametric MRI

Accurate quantification of uptake on PET images depends on accurate attenuation correction in reconstruction. Current MR-based attenuation correction methods for body PET use a fat and water map derived from a 2-echo Dixon MRI sequence in which bone is neglected. Ultrashort-echo-time or zero-echo-time (ZTE) pulse sequences can capture bone information. We propose the use of patient-specific multiparametric MRI consisting of Dixon MRI and proton-density-weighted ZTE MRI to directly synthesize pseudo-CT images with a deep learning model: we call this method ZTE and Dixon deep pseudo-CT (ZeDD CT). Methods: Twenty-six patients were scanned using an integrated 3-T time-of-flight PET/MRI system. Helical CT images of the patients were acquired separately. A deep convolutional neural network was trained to transform ZTE and Dixon MR images into pseudo-CT images. Ten patients were used for model training, and 16 patients were used for evaluation. Bone and soft-tissue lesions were identified, and the SUVmax was measured. The root-mean-squared error (RMSE) was used to compare the MR-based attenuation correction with the ground-truth CT attenuation correction. Results: In total, 30 bone lesions and 60 soft-tissue lesions were evaluated. The RMSE in PET quantification was reduced by a factor of 4 for bone lesions (10.24% for Dixon PET and 2.68% for ZeDD PET) and by a factor of 1.5 for soft-tissue lesions (6.24% for Dixon PET and 4.07% for ZeDD PET). Conclusion: ZeDD CT produces natural-looking and quantitatively accurate pseudo-CT images and reduces error in pelvic PET/MRI attenuation correction compared with standard methods

Differentiation between subcentimeter carcinomas and benign lesions using kinetic parameters derived from ultrafast dynamic contrast-enhanced breast MRI.

OBJECTIVES: This study aims to evaluate ultrafast DCE-MRI-derived kinetic parameters that reflect contrast agent inflow effects in differentiating between subcentimeter BI-RADS 4-5 breast carcinomas and benign lesions. METHODS: We retrospectively reviewed consecutive 3-T MRI performed from February to October 2017, during which ultrafast DCE-MRI was performed as part of a hybrid clinical protocol with conventional DCE-MRI. In total, 301 female patients with 369 biopsy-proven breast lesions were included. Ultrafast DCE-MRI was acquired continuously over approximately 60 s (temporal resolution, 2.7-7.1 s/phase) starting simultaneously with the start of contrast injection. Four ultrafast DCE-MRI-derived kinetic parameters (maximum slope [MS], contrast enhancement ratio [CER], bolus arrival time [BAT], and initial area under gadolinium contrast agent concentration [IAUGC]) and one conventional DCE-MRI-derived kinetic parameter (signal enhancement ratio [SER]) were calculated for each lesion. Wilcoxon rank sum test or Fishers exact test was performed to compare kinetic parameters, volume, diameter, age, and BI-RADS morphological descriptors between subcentimeter carcinomas and benign lesions. Univariate/multivariate logistic regression analyses were performed to determine predictive parameters for subcentimeter carcinomas. RESULTS: In total, 125 lesions (26 carcinomas and 99 benign lesions) were identified as BI-RADS 4-5 subcentimeter lesions. Subcentimeter carcinomas demonstrated significantly larger MS and SER and shorter BAT than benign lesions (p = 0.0117, 0.0046, and 0.0102, respectively). MS, BAT, and age were determined as significantly predictive for subcentimeter carcinoma (p = 0.0208, 0.0023, and < 0.0001, respectively). CONCLUSIONS: Ultrafast DCE-MRI-derived kinetic parameters may be useful in differentiating subcentimeter BI-RADS 4 and 5 carcinomas from benign lesions. KEY POINTS: • Ultrafast DCE-MRI can generate kinetic parameters, effectively differentiating breast carcinomas from benign lesions. • Subcentimeter carcinomas demonstrated significantly larger maximum slope and shorter bolus arrival time than benign lesions. • Maximum slope and bolus arrival time contribute to better management of suspicious subcentimeter breast lesions

Region of interest focused MRI to synthetic CT translation using regression and segmentation multi-task network

Objective:In MR-only clinical workflow, replacing CT with MR image is of advantage for workflow efficiency and reduces radiation to the patient. An important step required to eliminate CT scan from the workflow is to generate the information provided by CT via an MR image. In this work, we aim to demonstrate a method to generate accurate synthetic CT (sCT) from an MR image to suit the radiation therapy (RT) treatment planning workflow. We show the feasibility of the method and make way for a broader clinical evaluation.Approach:We present a machine learning method for sCT generation from zero-echo-time (ZTE) MRI aimed at structural and quantitative accuracies of the image, with a particular focus on the accurate bone density value prediction. The misestimation of bone density in the radiation path could lead to unintended dose delivery to the target volume and results in suboptimal treatment outcome. We propose a loss function that favors a spatially sparse bone region in the image. We harness the ability of the multi-task network to produce correlated outputs as a framework to enable localization of region of interest (RoI) via segmentation, emphasize regression of values within RoI and still retain the overall accuracy via global regression. The network is optimized by a composite loss function that combines a dedicated loss from each task.Main results:We have included 54 brain patient images in this study and tested the sCT images against reference CT on a subset of 20 cases. A pilot dose evaluation was performed on 9 of the 20 test cases to demonstrate the viability of the generated sCT in RT planning. The average quantitative metrics produced by the proposed method over the test set were-(a) mean absolute error (MAE) of 70 ± 8.6 HU; (b) peak signal-to-noise ratio (PSNR) of 29.4 ± 2.8 dB; structural similarity metric (SSIM) of 0.95 ± 0.02; and (d) Dice coefficient of the body region of 0.984 ± 0.Significance. We demonstrate that the proposed method generates sCT images that resemble visual characteristics of a real CT image and has a quantitative accuracy that suits RT dose planning application. We compare the dose calculation from the proposed sCT and the real CT in a radiation therapy treatment planning setup and show that sCT based planning falls within 0.5% target dose error. The method presented here with an initial dose evaluation makes an encouraging precursor to a broader clinical evaluation of sCT based RT planning on different anatomical regions.</p