Fast and accurate classification of echocardiograms using deep learning

Echocardiography is essential to modern cardiology. However, human interpretation limits high throughput analysis, limiting echocardiography from reaching its full clinical and research potential for precision medicine. Deep learning is a cutting-edge machine-learning technique that has been useful in analyzing medical images but has not yet been widely applied to echocardiography, partly due to the complexity of echocardiograms' multi view, multi modality format. The essential first step toward comprehensive computer assisted echocardiographic interpretation is determining whether computers can learn to recognize standard views. To this end, we anonymized 834,267 transthoracic echocardiogram (TTE) images from 267 patients (20 to 96 years, 51 percent female, 26 percent obese) seen between 2000 and 2017 and labeled them according to standard views. Images covered a range of real world clinical variation. We built a multilayer convolutional neural network and used supervised learning to simultaneously classify 15 standard views. Eighty percent of data used was randomly chosen for training and 20 percent reserved for validation and testing on never seen echocardiograms. Using multiple images from each clip, the model classified among 12 video views with 97.8 percent overall test accuracy without overfitting. Even on single low resolution images, test accuracy among 15 views was 91.7 percent versus 70.2 to 83.5 percent for board-certified echocardiographers. Confusional matrices, occlusion experiments, and saliency mapping showed that the model finds recognizable similarities among related views and classifies using clinically relevant image features. In conclusion, deep neural networks can classify essential echocardiographic views simultaneously and with high accuracy. Our results provide a foundation for more complex deep learning assisted echocardiographic interpretation.Comment: 31 pages, 8 figure

Clinical Utility of Echocardiography in Former Preterm Infants with Bronchopulmonary Dysplasia.

BackgroundThe clinical utility of echocardiography for the diagnosis of pulmonary vascular disease (PVD) in former preterm infants with bronchopulmonary dysplasia (BPD) is not established. Elevated pulmonary vascular resistance (PVR) rather than pulmonary artery pressure (PAP) is the hallmark of PVD. We evaluated the utility of echocardiography in infants with BPD in diagnosing pulmonary hypertension and PVD (PVR >3 Wood units × m2) assessed by cardiac catheterization.MethodsA retrospective single center study of 29 infants born ≤29 weeks of gestational age with BPD who underwent cardiac catheterization and echocardiography was performed. PVD was considered present by echocardiography if the tricuspid valve regurgitation jet peak velocity was >2.9 m/sec, post-tricuspid valve shunt systolic flow velocity estimated a right ventricular systolic pressure >35 mm Hg, or systolic septal flattening was present. The utility (accuracy, sensitivity, and positive predictive value [PPV]) of echocardiography in the diagnosis of PVD was tested. Subgroup analysis in patients without post-tricuspid valve shunts was performed. Echocardiographic estimations of right ventricular pressure, dimensions, function, and pulmonary flow measurements were evaluated for correlation with PVR.ResultsThe duration between echocardiography and cardiac catheterization was a median of 1 day (interquartile range, 1-4 days). Accuracy, sensitivity, and PPV of echocardiography in diagnosing PVD were 72%, 90.5%, and 76%, respectively. Accuracy, sensitivity, and PPV increased to 93%, 91.7%, and 100%, respectively, when infants with post-tricuspid valve shunts were excluded. Echocardiography had poor accuracy in estimating the degree of PAP elevation by cardiac catheterization. In infants without post-tricuspid valve shunts, there was moderate to good correlation between indexed PVR and right ventricular myocardial performance index (rho = 0.89, P = .005), systolic to diastolic time index (0.84, P < .001), right to left ventricular diameter ratio at end systole (0.66, P = .003), and pulmonary artery acceleration time (0.48, P = .05).ConclusionsEchocardiography performs well in screening for PVD in infants with BPD and may be diagnostic in the absence of a post-tricuspid valve shunt. However, cardiac catheterization is needed to assess the degree of PAP elevation and PVR. The diagnostic utility of echocardiographic measurements that correlate with PVR should be evaluated prospectively in this patient population

An improved classification approach for echocardiograms embedding temporal information

Cardiovascular disease is an umbrella term for all diseases of the heart. At present, computer-aided echocardiogram diagnosis is becoming increasingly beneficial. For echocardiography, different cardiac views can be acquired depending on the location and angulations of the ultrasound transducer. Hence, the automatic echocardiogram view classification is the first step for echocardiogram diagnosis, especially for computer-aided system and even for automatic diagnosis in the future. In addition, heart views classification makes it possible to label images especially for large-scale echo videos, provide a facility for database management and collection. This thesis presents a framework for automatic cardiac viewpoints classification of echocardiogram video data. In this research, we aim to overcome the challenges facing this investigation while analyzing, recognizing and classifying echocardiogram videos from 3D (2D spatial and 1D temporal) space. Specifically, we extend 2D KAZE approach into 3D space for feature detection and propose a histogram of acceleration as feature descriptor. Subsequently, feature encoding follows before the application of SVM to classify echo videos. In addition, comparison with the state of the art methodologies also takes place, including 2D SIFT, 3D SIFT, and optical flow technique to extract temporal information sustained in the video images. As a result, the performance of 2D KAZE, 2D KAZE with Optical Flow, 3D KAZE, Optical Flow, 2D SIFT and 3D SIFT delivers accuracy rate of 89.4%, 84.3%, 87.9%, 79.4%, 83.8% and 73.8% respectively for the eight view classes of echo videos

The application of KAZE features to the classification echocardiogram videos

In the computer vision field, both approaches of SIFT and SURF are prevalent in the extraction of scale-invariant points and have demonstrated a number of advantages. However, when they are applied to medical images with relevant low contrast between target structures and surrounding regions, these approaches lack the ability to distinguish salient features. Therefore, this research proposes a different approach by extracting feature points using the emerging method of KAZE. As such, to categorise a collection of video images of echocardiograms, KAZE feature points, coupled with three popular representation methods, are addressed in this paper, which includes the bag of words (BOW), sparse coding, and Fisher vector (FV). In comparison with the SIFT features represented using Sparse coding approach that gives 72% overall performance on the classification of eight viewpoints, KAZE feature integrated with either BOW, sparse coding or FV improves the performance significantly with the accuracy being 81.09%, 78.85% and 80.8% respectively. When it comes to distinguish only three primary view locations, 97.44% accuracy can be achieved when employing the approach of KAZE whereas 90% accuracy is realised while applying SIFT features

Deep Learning in Cardiology

The medical field is creating large amount of data that physicians are unable to decipher and use efficiently. Moreover, rule-based expert systems are inefficient in solving complicated medical tasks or for creating insights using big data. Deep learning has emerged as a more accurate and effective technology in a wide range of medical problems such as diagnosis, prediction and intervention. Deep learning is a representation learning method that consists of layers that transform the data non-linearly, thus, revealing hierarchical relationships and structures. In this review we survey deep learning application papers that use structured data, signal and imaging modalities from cardiology. We discuss the advantages and limitations of applying deep learning in cardiology that also apply in medicine in general, while proposing certain directions as the most viable for clinical use.Comment: 27 pages, 2 figures, 10 table

The application of KAZE features to the classification echocardiogram videos

In the computer vision field, both approaches of SIFT and SURF are prevalent in the extraction of scale-invariant points and have demonstrated a number of advantages. However, when they are applied to medical images with relevant low contrast between target structures and surrounding regions, these approaches lack the ability to distinguish salient features. Therefore, this research proposes a different approach by extracting feature points using the emerging method of KAZE. As such, to categorise a collection of video images of echocardiograms, KAZE feature points, coupled with three popular representation methods, are addressed in this paper, which includes the bag of words (BOW), sparse coding, and Fisher vector (FV). In comparison with the SIFT features represented using Sparse coding approach that gives 72% overall performance on the classification of eight viewpoints, KAZE feature integrated with either BOW, sparse coding or FV improves the performance significantly with the accuracy being 81.09%, 78.85% and 80.8% respectively. When it comes to distinguish only three primary view locations, 97.44% accuracy can be achieved when employing the approach of KAZE whereas 90% accuracy is realised while applying SIFT features

A fused deep learning architecture for viewpoint classification of echocardiography

This study extends the state of the art of deep learning convolutional neural network (CNN) to the classification of video images of echocardiography, aiming at assisting clinicians in diagnosis of heart diseases. Specifically, the architecture of neural networks is established by embracing hand-crafted features within a data-driven learning framework, incorporating both spatial and temporal information sustained by the video images of the moving heart and giving rise to two strands of two-dimensional convolutional neural network (CNN). In particular, the acceleration measurement along the time direction at each point is calculated using dense optical flow technique to represent temporal motion information. Subsequently, the fusion of both networks is conducted via linear integrations of the vectors of class scores obtained from each of the two networks. As a result, this architecture maintains the best classification results for eight viewpoint categories of echo videos with 92.1% accuracy rate whereas 89.5% is achieved using only single spatial CNN network. When concerning only three primary locations, 98% of accuracy rate is realised. In addition, comparisons with a number of well-known hand-engineered approaches are also performed, including 2D KAZE, 2D KAZE with Optical Flow, 3D KAZA, Optical Flow, 2D SIFT and 3D SIFT, which delivers accuracy rate of 89.4%, 84.3%, 87.9%, 79.4%, 83.8% and 73.8% respectively