International audienceIn this paper we propose a method for automatic wrangling of missing or noisy acquisition plane information of cardiac magnetic resonance images in order to simplify case filtering and image lookup in large collections of cardiac data. To recognize standard cardiac planes we use features based on image miniatures combined with a decision forest classifier. We show that augmenting the dataset with a set of nondestructive transformations can improve classification accuracy. Our approach compares favorably to the state of the art while requiring fewer manual annotations

Ayache, Nicholas

Criminisi, Antonio

Lee, Daniel C.

Margeta, Jan

HAL-Rennes 1

Recognizing cardiac magnetic resonance acquisition planes

INRIA a CCSD electronic archive server

HAL Id: hal-01009952https://hal.inria.fr/hal-01009952Submitted on 19 Jun 2014HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.Recognizing cardiac magnetic resonance acquisitionplanesJan Margeta, Antonio Criminisi, Daniel C. Lee, Nicholas AyacheTo cite this version:Jan Margeta, Antonio Criminisi, Daniel C. Lee, Nicholas Ayache. Recognizing cardiac magneticresonance acquisition planes. MIUA - Medical Image Understanding and Analysis Conference - 2014,Reyes-Aldasoro, Constantino Carlos and Slabaugh, Gregory, Jul 2014, London, United Kingdom.￿hal-01009952￿MARGETA, CRIMINISI, LEE, AYACHE: CARDIAC ACQUISITION PLANE RECOGNITION 1Recognizing cardiac magnetic resonanceacquisition planesJan Margeta1jan@kardio.meAntonio Criminisi2antcrim@microsoft.comDaniel C Lee3dlee@northwestern.eduNicholas Ayache1nicholas.ayache@inria.fr1 Asclepios, Inria Sophia Antipolis,2004 Route des LuciolesSophia Antipolis, France2 Machine learning and perception groupMicrosoft Research,21 Station Rd, Cambridge, UK3 Feinberg Cardiovascular ResearchInstitute, Northwestern University,Feinberg School of Medicine,Chicago, USAAbstractIn this paper we propose a method for automatic wrangling of missing or noisy ac-quisition plane information of cardiac magnetic resonance images in order to simplifycase filtering and image lookup in large collections of cardiac data. To recognize stan-dard cardiac planes we use features based on image miniatures combined with a decisionforest classifier. We show that augmenting the dataset with a set of nondestructive trans-formations can improve classification accuracy. Our approach compares favorably to thestate of the art while requiring fewer manual annotations.1 IntroductionCardiac magnetic resonance image acquisitions are mostly performed along several obliqueaxes with respect to the structures of the heart instead of traditional body planes (coronal,axial and sagital) to better visualize and quantify different aspects of the heart. Knowledgeof these planes is also essential for data collection organization, selecting adapted image pro-cessing algorithms, grouping of related slices into volumetric image stacks, filtering of casesbased on completeness for a clinical study and to respond with the most relevant acquisitionplane in content based image retrieval.This orientation information is often in some way encoded within the DICOM imagetags: Series Description (0008,103E) and Protocol Name (0018,1030). The way to encodethis view information is however not standardized, operator errors are present or such in-formation is often completely missing. Searching through large databases to handpick de-sired views from cases in the collection is therefore tedious. Previous work in this field hasconcentrated mainly on real-time recognition of cardiac planes in echography acquisitions.These methods are based on appearance models, registration or part detection and requireadditional information to train view, part [6] or landmark specific detectors [10]. Thereforec© 2014. The copyright of this document resides with its authors.It may be distributed unchanged freely in print or electronic forms.(a) 2CH (b) 3CH (c) 4CH (d) LVOT2CH3CH4CH(e) SAXFigure 1: Examples of the main left ventricular long and short axis cardiac MR views.any new view will require these extra annotations to be made. Otey et al. [5] on the otherhand addressed this problem by using view labels only from gradient based image features.1.1 Cardiac planes of acquisitionOptimal cardiac planes depend on global positioning of the heart in the thorax. This is morevertical in young individuals and more diaphragmatic in elderly. Imaging in standard car-diac planes ensures efficient coverage of relevant cardiac territories and enables comparisonsacross modalities, thus enhancing patient care and cardiovascular research. A good overviewof standard cardiac acquisition planes can be found in [8].Left ventricular long axis acquisition planes. These are usually acquired as 2D or cine2D+t stack. The 2-chamber, 3-chamber, and 4-chamber views (figs. 1(a) to 1(c)) are usedto visualize different regions of the left atrium, mitral valve apparatus, and left ventricle.The 3-chamber and left ventricular outflow tract (fig. 1(d)) views provide visualization ofthe aortic root from two orthogonal planes. The 4-chamber view enables visualization of thetricuspid valve and right atrium.Short axis acquisition planes. Short axis slices (fig. 1(e)) are oriented parallel to the mitralvalve ring. These are acquired regularly spaced from the cardiac base to the apex of theheart, often as a cine 3D+t stack. These views are excellent for reproducible volumetricmeasurements or radial cardiac motion analysis but their use is limited in atrio-ventricularinterplay or valvular disease study.2 MethodWe propose automatic cardiac view recognition pipeline (Fig. 2(a)) that learns to recognizethe acquisition planes directly from cardiac magnetic resonance images by combining imageminiatures with classification forests. These are trained in a supervised fashion to automat-ically select the relevant features. To augment the training dataset we jitter the images withsmall translations, rotations and scales. Apart from the view annotation of the training set,no other information is necessary to classify the views.2.1 Slice normals from DICOMFor images where the DICOM orientation (0020,0037) tag is present we can use it to predictthe cardiac image acquisition plane. Similarly to [10] we extract image acquisition planenormal vectors as a cross-product of the two image orientation vectors specified in the tagand use this three-dimensional vector feature vector to describe each image (See Fig. 2(b)).Crop central squareClassification forestJitter (translate, rotate, scale) and crop training slicesMiniatures and intensity poolingTrainingTesting2CH3CH4CHSAX2CH3CH4CHSAX2CH3CH4CHSAXLVOTLVOTLVOT(a) Discriminative pixels from image miniatures are chosen from a random poolas features for a classification forest. We jitter the training dataset to improverobustness to differences in the acquisitions without the need for extra data.(b) DICOM plane normals fordifferent cardiac views.128x128 (4x4 tiles)40x40 20x2032x32 32x32miniaturesmin & max poolImage tiles(c) Image feature maps as downsampled im-ages and tile intensity minima and maxima.Figure 2: Our view recognition pipeline (above) and used features (below).2.2 Image-based featuresWhen the orientation tag is not present we rely on the image intensity information only. Tocapture differences between planes of acquisition, we extract a set of image based features.To allow use of this for method for all of the above mentioned applications, we use 2D imageslices individually rather than 3D or 3D+ t volumes.Preprocessing Our only preprocessing steps are image central square extraction, resam-pling to 128x128 pixels and linear rescaling of the intensities to range between 0 and 255.Pooled image miniatures Radiological images are mostly acquired with the object of in-terest in the center. Therefore some rough alignment of structures can be expected (Seefig. 3). Image intensity samples at fixed locations can therefore provide strong clues aboutthe position of different tissues (e.g. symmetric dark lungs or bright cavity in the center).It has been shown before [9] that significantly down-sampled miniatures can be used2CH 4CH3CH LVOTSAXAp SAXA SAXM SAXBSAXSBFigure 3: Image miniature averages for each cardiac view after cropping the central square.Short axis view means are displayed from the apex SAXAp to above base slices SAXSB.Distinct patterns of the bright blood pool or dark lungs can be observed for each view.for such image recognition. In this paper we subsample the cropped centers to two fixedsizes (20x20 and 40x40 pixels). We also create another set of pooled image miniatures fromeach cropped center by dividing the image into non-overlapping 4x4 tiles and computingintensity minimum and maximum in each of these tiles (fig. 2(c)). As in [4], pooling canadd invariance to small image translations and rotations (whose effect is within the tile size).We used directly pixel values from these miniatures sampled at random positions as features.Each image can be seen as a data point in this high-dimensional feature space.2.3 Decision forest classifierRandom decision forest [1] is an ensemble classification method that consists of a set of bi-nary decision trees. This method is computationally efficient and allows automatic selectionof relevant features for the prediction. The tree structure is optimized by recursively parti-tioning the collection of data points X into the left or right branches such that points withdifferent labels get separated while the same label points are grouped together. At each nodeof the tree a feature from a randomly drawn subset of all features is chosen such that impu-rity I in both branches is minimized. We weight samples from the under-represented classesmore and we set sample weights inversely proportional to class probabilities and normalizethem such that sum wle f t +wright to equal to one at each node.I(X ,θ) = wle f tH(Xle f t)+wrightH(Xright) (1)H is weighted entropy and Xle f t and Xright are point subsets falling to either the left or theright branch, based on the tested feature value and threshold. Similarly, wle f t , wright are sumsof sample weights at each branch and wc is sum of weights for a particular class c.H(X) =−∑cwclog(wc) (2)Only a random subset of features (i.e. single pixel values at 64 different locations of theminiatures) is tested at each node of each tree and a simple threshold on this value is used todivide the data points into the left or the right partition. This helps to make the trees in theforest different from each other which leads to better generalization.When classifying a new image, features chosen at the training stage are extracted and theimage is passed through the decisions of the forest (fixed in the training phase) to reach aset of leaves. Class distributions of the reached leaves are averaged across the forest and themost probable label is selected as the image view.2.4 Augmenting the dataset with jitteringOur dataset contains in total 960 image slices from 100 cardiac patients (SAX:540, 4CH:140, 2CH: 112, 3CH: 107, LVOT: 9). This might seem rather modest compared to recentimage recognition work e.g. [4] where deep models are trained with 15 million images. Theobject of interest is in general placed in the image center, however deviations exist. Toaccount for these differences we expand the training set with extra images by applying a setof transformations of the original training images. Similarly to [4] these are translations (allshifts in x and y between -10 and 10 pixels for a 5x5 grid), but we also add rotations (anglesbetween -10 and 10 degrees with 20 steps) and scales (1-1.4 zoom factor with 8 steps whilekeeping the same image size) resulting in a total of 51894 training images. Note that thisextra expense is mainly at the training time. The test time is almost unaffected except thatnow a deeper forest can be learnt.0 20 40 60 80 100 120 140 160Number of trees8284868890Classification accuracy(a) The effect of the number of trees.2CH4CH3CHLVOT SAX2CH4CH3CHLVOTSAX7 0 0 0 10 14 0 0 02 2 12 0 01 0 0 0 01 0 0 0 560.00.10.20.30.40.50.60.70.80.91.0(b) Confusion matrix from one fold using im-age features only.Figure 4: We selected the optimal parameters via cross validation. Despite the sampleweighting the LVOT view recognition suffered from being underrepresented.3 ValidationDatasets We validated this method on a dataset of 100 patients from multi-center studyon post myocardial infarction hearts DETERMINE [3]. These consist of steady state freeprecession magnetic resonance acquisitions.Results We ran a 25-fold cross validation by dividing the dataset on patient identifier toprevent biasing our results. This means that images from the same patient (despite beinga different view) and therefore the same acquisition could never appear in the training andtesting set at the same time. Here we present results for the merged short axis view labels byusing features extracted exclusively either from image-based features or from DICOM sliceorientation information.Image features only DICOM orientationNo jitter With jitter features onlyAccuracy 87.1 ± 1.0% 90.2 ± 1.6% 98.8 ± 5.4%Increasing the number of trees from 10 to 160 improves the prediction accuracy, however atthe expense of increased training and classification computational cost . Beyond 80 trees theforest reaches a plateau with no further benefit (See Fig. 4(a)).ConclusionIn this paper we presented an automatic cardiac view recognition technique from magneticresonance images based on image appearance and DICOM plane orientation information.The DICOM orientation based features can be used for our population with confidence whenthis information is present. When this is not the case, the pure image information methodcan be used instead. It is achieving precision on par or better with pure image based resultsof [10] while being much simpler and not requiring to train any landmark detectors.By jittering the dataset with extra translations, rotations and scales we observed notice-able benefits to view recognition. Extra jittering steps might be useful to improve robustnessto intensity differences between different acquisitions. If enough examples are collected, thismethod can be simply extended to other pathology specific views such as in congenital heartdiseases and acquisition sequences different from SSFP.We plan to use our method to organize collections of images, semantic image retrievaland parsing of medical literature.Acknowledgments. This work was partly supported by Microsoft Research through its PhD Schol-arship Programme and ERC grant MedYMA. We used data and infrastructure made available throughthe Cardiac Atlas Project (www.cardiacatlas.org) [2][3]. This work uses scikit-learn toolkit[7].References[1] Leo Breiman. Random forests-random features. pages 1–29, 1999.[2] CG Fonseca, M Backhaus, DA Bluemke, RD Britten, JD Chung, BR Cowan, ID Dinov, JP Finn,PJ Hunter, AH Kadish, DC Lee, JAC Lima, P Medrano-Gracia, K Shivkumar, A Suinesiaputra,W Tao, and AA Young. The Cardiac Atlas Project- an Imaging Database for ComputationalModeling and Statistical Atlases of the Heart. Bioinformatics, 27(16):2288–2295, 2011.[3] Alan H Kadish, David Bello, J Paul Finn, Robert O Bonow, Andi Schaechter, Haris Subacius,Christine Albert, James P Daubert, Carissa G Fonseca, and Jeffrey J Goldberger. Rationale anddesign for the Defibrillators to Reduce Risk by Magnetic Resonance Imaging Evaluation (DE-TERMINE) trial. Journal of cardiovascular electrophysiology, 20(9):982–7, September 2009.ISSN 1540-8167.[4] Alex Krizhevsky, I Sutskever, and GE Hinton. ImageNet Classification with Deep ConvolutionalNeural Networks. In Advances in Neural Information Processing Systems, 2012.[5] M Otey, Jinbo Bi, S Krishna, and Bharat Rao. Automatic view recognition for cardiac ultrasoundimages. In International Workshop on Computer Vision for Intravascular and Intracardiac Imag-ing, pages 187–194, 2006.[6] J. H. Park, S. K. Zhou, C. Simopoulos, J. Otsuki, and D. Comaniciu. Automatic Cardiac ViewClassification of Echocardiogram. IEEE 11th International Conference on Computer Vision,pages 1–8, 2007.[7] F Pedregosa, G Varoquaux, A Gramfort, V Michel, B Thirion, O Grisel, M Blondel, P Pretten-hofer, R Weiss, V Dubourg, J Vanderplas, A Passos, D Cournapeau, M Brucher, M Perrot, andE Duchesnay. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research,12:2825–2830, 2011.[8] A M Taylor and J Bogaert. Cardiovascular MR Imaging Planes and Segmentation. In Jan Bogaert,Steven Dymarkowski, Andrew M Taylor, and Vivek Muthurangu, editors, Clinical Cardiac MRISE - 333, Medical Radiology, pages 93–107. Springer Berlin Heidelberg, 2012. ISBN 978-3-642-23034-9.[9] Antonio Torralba, Rob Fergus, and William T Freeman. 80 Million Tiny Images: a Large DataSet for Nonparametric Object and Scene Recognition. IEEE transactions on pattern analysis andmachine intelligence, 30(11):1958–70, November 2008. ISSN 1939-3539.[10] Y Zhou, Zhigang Peng, and XS Zhou. Automatic view classification for cardiac MRI. In 9th IEEEInternational Symposium on Biomedical Imaging (ISBI), pages 1771–1774, Barcelona, 2012.IEEE. ISBN 9781457718588.

https://hal.inria.fr/hal-01009952/document

Recognizing cardiac magnetic resonance acquisition planes

Abstract

Similar works

Full text

Available Versions

HAL-Rennes 1

INRIA a CCSD electronic archive server