Automating Carotid Intima-Media Thickness Video Interpretation with Convolutional Neural Networks

Cardiovascular disease (CVD) is the leading cause of mortality yet largely preventable, but the key to prevention is to identify at-risk individuals before adverse events. For predicting individual CVD risk, carotid intima-media thickness (CIMT), a noninvasive ultrasound method, has proven to be valuable, offering several advantages over CT coronary artery calcium score. However, each CIMT examination includes several ultrasound videos, and interpreting each of these CIMT videos involves three operations: (1) select three end-diastolic ultrasound frames (EUF) in the video, (2) localize a region of interest (ROI) in each selected frame, and (3) trace the lumen-intima interface and the media-adventitia interface in each ROI to measure CIMT. These operations are tedious, laborious, and time consuming, a serious limitation that hinders the widespread utilization of CIMT in clinical practice. To overcome this limitation, this paper presents a new system to automate CIMT video interpretation. Our extensive experiments demonstrate that the suggested system significantly outperforms the state-of-the-art methods. The superior performance is attributable to our unified framework based on convolutional neural networks (CNNs) coupled with our informative image representation and effective post-processing of the CNN outputs, which are uniquely designed for each of the above three operations.Comment: J. Y. Shin, N. Tajbakhsh, R. T. Hurst, C. B. Kendall, and J. Liang. Automating carotid intima-media thickness video interpretation with convolutional neural networks. CVPR 2016, pp 2526-2535; N. Tajbakhsh, J. Y. Shin, R. T. Hurst, C. B. Kendall, and J. Liang. Automatic interpretation of CIMT videos using convolutional neural networks. Deep Learning for Medical Image Analysis, Academic Press, 201

An end-to-end framework for intima media measurement and atherosclerotic plaque detection in the carotid artery

Background and objectives: The detection and delineation of atherosclerotic plaque are usually manually performed by medical experts on the carotid artery. Evidence suggests that this manual process is subject to errors and has a large variability between experts, equipment, and datasets. This paper proposes a robust end-to-end framework for automatic atherosclerotic plaque detection. Methods: The proposed framework is composed of: (1) a semantic segmentation model based on U-Net, with EfficientNet as the backbone, that obtains a segmentation mask with the carotid intima-media region; and (2) a convolutional neural network designed using Bayesian optimization that simultaneously performs a regression to get the average and maximum carotid intima media thickness, and a classification to determine the presence of plaque. Results: Our approach improves the state-of-the-art in both co and bulb territories in the REGICOR database, with more than 8000 images, while providing predictions in real-time. The correlation coefficient was 0.89 in the common carotid artery and 0.74 for bulb region, and the F1 score for atherosclerotic plaque detecting was 0.60 and 0.59, respectively. The experimentation carried out includes a comparison with other fully automatic methods for carotid intima media thickness estimation found in the literature. Additionally, we present an extensive experimental study to evaluate the robustness of our proposal, as well as its suitability and efficiency compared to different versions of the framework. Conclusions: The proposed end-to-end framework significantly improves the automatic characterization of atherosclerotic plaque. The generation of the segmented mask can be helpful for practitioners since it allows them to evaluate and interpret the model's results by visual inspection. Furthermore, the proposed framework overcomes the limitations of previous research based on ad-hoc post-processing, which could lead to overestimations in the case of oblique forms of the carotid artery

Bimodal automated carotid ultrasound segmentation using geometrically constrained deep neural networks

For asymptomatic patients suffering from carotid stenosis, the assessment of plaque morphology is an important clinical task which allows monitoring of the risk of plaque rupture and future incidents of stroke. Ultrasound Imaging provides a safe and non-invasive modality for this, and the segmentation of media-adventitia boundaries and lumen-intima boundaries of the Carotid artery form an essential part in this monitoring process. In this paper, we propose a novel Deep Neural Network as a fully automated segmentation tool, and its application in delineating both the media-adventitia boundary and the lumen-intima boundary. We develop a new geometrically constrained objective function as part of the Network's Stochastic Gradient Descent optimisation, thus tuning it to the problem at hand. Furthermore, we also apply a bimodal fusion of amplitude and phase congruency data proposed by us in previous work, as an input to the network, as the latter provides an intensity-invariant data source to the network. We finally report the segmentation performance of the network on transverse sections of the carotid. Tests are carried out on an augmented dataset of 81,000 images, and the results are compared to other studies by reporting the DICE coefficient of similarity, modified Hausdorff Distance, sensitivity and specificity. Our proposed modification is shown to yield improved results on the standard network over this larger dataset, with the advantage of it being fully automated. We conclude that Deep Neural Networks provide a reliable trained manner in which carotid ultrasound images may be automatically segmented, using amplitude data and intensity invariant phase congruency maps as a data source

Can multiple segmentation methods enhance deep learning networks generalization? A novel hybrid learning paradigm

Deep learning methods are the state-of-the-art for medical imaging segmentation tasks. Still, numerous segmentation algorithms based on heuristic-based methods have been proposed with exceptional results. To validate segmentation algorithms, manual annotations are typically considered as ground truth. However, manual annotations often suffer from inter/intra-operator variability and can also be occasionally inaccurate, especially when considering time-consuming and precise tasks. A sample case is the manual delineation of the lumen-intima (LI) and media-adventitia (MA) borders for intima-media thickness (IMT) measurement in B-mode ultrasound images. In this work, a novel hybrid learning paradigm which combines manual segmentations with the automatic segmentation of a dynamic programming technique for ground truth determination is presented. A profile consensus strategy is proposed to construct the hybrid ground truth. Two open-source datasets (n=2576) were employed for training four deep learning networks using the hybrid learning paradigm and three single source training targets as a comparison. The pipeline was fixed across the four tests and included a Faster R-CNN detection network to locate the carotid artery and then subsequent division into patches which were segmented using a UNet. The validation of the results was performed on an external test set comparing the predictions of the four different models to the annotations of three independent manual operators. The hybrid learning paradigm showed the best overall segmentation results (Dice=0.907±0.037, p<0.001) and demonstrated an exceptional correlation between the mean of three operators and the automatic measure (ICC(2,1)=0.958), demonstrating how the incorporation of heuristic-based segmentation methods within the learning paradigm of a deep neural network can enhance and improve final segmentation performance results

Exploring the Impact of Learning Paradigms on Network Generalization: A Multi-Center IMT Study

The intima-media thickness (IMT) is an important parameter for evaluating cardiovascular disease risk and progression and can be extracted from B-mode longitudinal ultrasound images of the carotid artery. Despite its clinical significance, inter- and intra-operator variability in IMT measurement is a challenge due to subjective factors. Therefore, automatic and semi-automatic approaches based on heuristic methods and deep neural networks have been proposed to reduce the variability in IMT measurement. However, the inter- and intra- operator variability still remains an issue as it affects the quality and diversity of ground truth (GT) data used for training deep learning models. In this study, the authors evaluate the performance of different learning paradigms using different GTs on a multi-center IMT dataset. A recent segmentation network, ConvNeXt, is trained on a dataset of 2576 B-mode longitudinal ultrasound images of the carotid artery, using different GT annotations and learning paradigms. The method is then tested on an external dataset of 448 images from four different centers for which three manual segmentations were available. The results show how the use of different GT annotations and learning paradigms can enhance the generalization ability of deep learning models, demonstrating the importance of selecting appropriate GT data and learning strategies in achieving robust and reliable solutions. The study highlights the significance of incorporating heuristic methods in the training process of deep learning models to enhance the accuracy and consistency of IMT measurement, thus enabling more precise cardiovascular disease risk assessment

Multimodality carotid plaque tissue characterization and classification in the artificial intelligence paradigm: a narrative review for stroke application

Cardiovascular disease (CVD) is one of the leading causes of morbidity and mortality in the United States of America and globally. Carotid arterial plaque, a cause and also a marker of such CVD, can be detected by various non-invasive imaging modalities such as magnetic resonance imaging (MRI), computer tomography (CT), and ultrasound (US). Characterization and classification of carotid plaque-type in these imaging modalities, especially into symptomatic and asymptomatic plaque, helps in the planning of carotid endarterectomy or stenting. It can be challenging to characterize plaque components due to (I) partial volume effect in magnetic resonance imaging (MRI) or (II) varying Hausdorff values in plaque regions in CT, and (III) attenuation of echoes reflected by the plaque during US causing acoustic shadowing. Artificial intelligence (AI) methods have become an indispensable part of healthcare and their applications to the non-invasive imaging technologies such as MRI, CT, and the US. In this narrative review, three main types of AI models (machine learning, deep learning, and transfer learning) are analyzed when applied to MRI, CT, and the US. A link between carotid plaque characteristics and the risk of coronary artery disease is presented. With regard to characterization, we review tools and techniques that use AI models to distinguish carotid plaque types based on signal processing and feature strengths. We conclude that AI-based solutions offer an accurate and robust path for tissue characterization and classification for carotid artery plaque imaging in all three imaging modalities. Due to cost, user-friendliness, and clinical effectiveness, AI in the US has dominated the most

Deep learning for the detection and characterization of the carotid artery in ultrasound imaging

Treballs finals del Màster de Fonaments de Ciència de Dades, Facultat de matemàtiques, Universitat de Barcelona, Any: 2018, Tutor: Laura Igual Muñoz[en] Atherosclerosis is the main process causing most Cardio Vascular (CV) diseases. The measurement of Intima Media Thickness (IMT) in artery ultrasound images can be used to detect the presence of atherosclerotic plaques, which may appear in several territories of the artery. Moreover, it is well known that disruption of atherosclerotic plaque plays a crucial role in the pathogenesis of CV events. Several works have tried to automatize the detection of the IMT and the classification of the plaque by its composition. Traditionally, the methods used in the literature are semi-automatic. Furthermore, very little work has been done using Deep Learning approaches in order to solve this problems. In this thesis, we explore the effectiveness of Deep Learning techniques in attempting to automatize and improve the diagnosis of atheroma plaques. To achieve so we tackle the following problems: ultrasound image segmentation and plaque tissue classification. The techniques applied in this work are the following. For the segmentation of the common carotid artery IMT we replicate a state of the art Fully Convolutional Network approach and explore the implementation of a trained network to another dataset. Regarding the plaque classification problem, we explore the performance of Convolutional Neural Networks as well with two baseline methods. These techniques are applied on two datasets: REGICOR and NEFRONA. These datasets are provided by two research groups of IMIM and IRBLleida in collaboration in a larger project with the UB. A data exploration analysis is also presented on the patient’s data of NEFRONA to justify the importance of detecting the atherosclerotic plaques and thus the techniques we explore

Deep learning-based carotid media-adventitia and lumen-intima boundary segmentation from three-dimensional ultrasound images

Purpose: Quantification of carotid plaques has been shown to be important for assessing as well as monitoring the progression and regression of carotid atherosclerosis. Various metrics have been proposed and methods of measurements ranging from manual tracing to automated segmentations have also been investigated. Of those metrics, quantification of carotid plaques by measuring vessel-wall-volume (VWV) using the segmented media-adventitia (MAB) and lumen-intima (LIB) boundaries has been shown to be sensitive to temporal changes in carotid plaque burden. Thus, semi-automatic MAB and LIB segmentation methods are required to help generate VWV measurements with high accuracy and less user interaction. Methods: In this paper, we propose a semiautomatic segmentation method based on deep learning to segment the MAB and LIB from carotid three-dimensional ultrasound (3DUS) images. For the MAB segmentation, we convert the segmentation problem to a pixel-by-pixel classification problem. A dynamic convolutional neural network (Dynamic CNN) is proposed to classify the patches generated by sliding a window along the norm line of the initial contour where the CNN model is fine-tuned dynamically in each test task. The LIB is segmented by applying a region-of-interest of carotid images to a U-Net model, which allows the network to be trained end-to-end for pixel-wise classification. Results: A total of 144 3DUS images were used in this development, and a threefold cross-validation technique was used for evaluation of the proposed algorithm. The proposed algorithm-generated accuracy was significantly higher than the previous methods but with less user interactions. Comparing the algorithm segmentation results with manual segmentations by an expert showed that the average Dice similarity coefficients (DSC) were 96.46 ± 2.22% and 92.84 ± 4.46% for the MAB and LIB, respectively, while only an average of 34 s (vs 1.13, 2.8 and 4.4 min in previous methods) was required to segment a 3DUS image. The interobserver experiment indicated that the DSC was 96.14 ± 1.87% between algorithm-generated MAB contours of two observers\u27 initialization. Conclusions: Our results showed that the proposed carotid plaque segmentation method obtains high accuracy and repeatability with less user interactions, suggesting that the method could be used in clinical practice to measure VWV and monitor the progression and regression of carotid plaques