Akbar, Shazia

Jordan, Lee

McKenna, Stephen J.

Thompson, Alastair M.

English

Tumor localization is an important component of histopathology image analysis; it has yet to be reliably automated for breast cancer histopathology. This paper investigates the use of superpixel classification to localize tumor regions. A superpixel representation retains information about visual structures such as cellular compartments, connective tissue, lumen and fatty tissue without having to commit to semantic segmentation at this level. In order to localize tumor in large images, a rotation invariant spatial pyramid representation is proposed using bags-of-superpixels. The method is evaluated on expert-annotated oestrogen-receptor stained TMA spots and compared to other superpixel classification techniques. Results demonstrate that it performs favorably

University of Dundee Online Publications

                                                                    University of DundeeTumor localization in tissue microarrays using rotation invariant superpixel pyramidsAkbar, Shazia; Jordan, Lee; Thompson, Alastair M.; McKenna, Stephen J.Published in:2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI 2015)DOI:10.1109/ISBI.2015.7164111Publication date:2015Document VersionPeer reviewed versionLink to publication in Discovery Research PortalCitation for published version (APA):Akbar, S., Jordan, L., Thompson, A. M., & McKenna, S. J. (2015). Tumor localization in tissue microarrays usingrotation invariant superpixel pyramids. In 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI2015) (pp. 1292-1295). IEEE. https://doi.org/10.1109/ISBI.2015.7164111General rightsCopyright and moral rights for the publications made accessible in Discovery Research Portal are retained by the authors and/or othercopyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated withthese rights. • Users may download and print one copy of any publication from Discovery Research Portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain. • You may freely distribute the URL identifying the publication in the public portal.Take down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Download date: 03. Sep. 2023TUMOR LOCALIZATION IN TISSUE MICROARRAYS USING ROTATION INVARIANTSUPERPIXEL PYRAMIDSShazia Akbar?, Lee Jordan†, Alastair M. Thompson‡, Stephen J. McKenna??School of Computing, University of Dundee, Dundee, UK†Pathology, Ninewells Hospital, Dundee, UK‡Department of Surgical Oncology, MD Anderson Cancer Center, Houston, United StatesABSTRACTTumor localization is an important component of histopathol-ogy image analysis; it has yet to be reliably automated forbreast cancer histopathology. This paper investigates the useof superpixel classification to localize tumor regions. A su-perpixel representation retains information about visual struc-tures such as cellular compartments, connective tissue, lumenand fatty tissue without having to commit to semantic seg-mentation at this level. In order to localize tumor in largeimages, a rotation invariant spatial pyramid representation isproposed using bags-of-superpixels. The method is evaluatedon expert-annotated oestrogen-receptor stained TMA spotsand compared to other superpixel classification techniques.Results demonstrate that it performs favorably.Index Terms— tumor classification, superpixels, spatialbag-of-words, rotation invariant spatial pyramid1. INTRODUCTIONTissue microarray (TMA) spots such as shown in Fig. 1(a) en-able relatively high-throughput inspection of various cancersfor multiple molecular markers [1]. They are used for val-idation of diagnostic markers in annotated clinical samples,testing new antibodies and probes, or determining optimalstaining conditions. The growth of tissue banks has lead toan increased workload for pathology experts, exceeding themanual skills available. Furthermore, expert pathology re-view suffers from inter- and intra-observer variations.Tumor localization is an important component of histo-pathology image analysis such as immunohistochemical(IHC) scoring and has not been reliably automated in breastTMAs. It is challenging because breast tissue has complexvisual structure; currently no detailed guidelines exist on howto visually identify tumor regions.One approach to automate tumor analysis is to first explic-itly detect and segment cells or subcellular components andthen build a feature representation based on these detectedobjects [3, 4]. However, in the images considered here, cel-lular compartment boundaries other than those of nuclei arenot often apparent. Nuclei detection is non-trivial because of(a) (b)Fig. 1. (a) A tissue microarray spot with annotated tumor re-gions. (b) An image patch (top) and a SLIC [2] superpixel im-age with each superpixel rendered as the average RGB valueof the pixels contained in it (bottom).the different cell types, varied appearance, and often heavilyclustered nuclei. An alternative approach is to treat the prob-lem as one of texture segmentation, without explicit detectionof objects [5,6]. However, statistical texture descriptors com-puted directly from the pixels may fail to capture structuralaspects of tissue.We investigate features based on superpixels to localizetumor regions. Fig. 1(b) shows a superpixel representation.The intuition is that a superpixel-based representation canretain information about visual structures such as cellularcompartments, connective tissue, lumen and fatty tissue with-out having to commit to semantic segmentation at this level.Superpixels have previously been applied in histopathologyfor cell nuclei segmentation [7] and tumor classification [8].Here, we use features from superpixels in a Bag-of-Words(BoW) framework, called Bags-of-Superpixels (BoS).We make the following contributions in this paper. (i) In-corporation of spatial information in BoS (iii) extension of thespatial pyramid [9] to build a Rotation Invariant SuperpixelPyramid (RISP) (iv) an adaptation of the autocorrelogram in-corporating superpixels. Tumor localization is formulated asTo appear in the Proceedings of 2015 IEEE International Symposium on Biomedical Imaging (ISBI 2015).Appearance Mean RGB, greyscale mean and variance,13 Haralick texture features (f1 − f13).Geometry Compactness, eccentricity, area, perimeter.Neighbors Number of immediate superpixel neigh-bors, variance of superpixel perimetershared with neighboring superpixels.Table 1. Features extracted from each superpixel.a superpixel classification problem and is evaluated on expert-annotated oestrogen-receptor stained TMA spots.2. METHODS2.1. Superpixel FeaturesA superpixel image representation is an over-segmentation inwhich similar pixels are grouped into perceptually consistentunits, regularized to be compact and of similar size. We useSLIC [2] to construct superpixels as it is computationally ef-ficient and retains tissue structure. The number of superpixelsis assigned such that the area of a single superpixel rarely ex-ceeds the area of a cell nucleus. Many cell nuclei are assignedtwo or more superpixels. An example SLIC representation ofa breast TMA patch is shown in Fig. 1(b) in which the aver-age superpixel area is 223 pixels (σ = 33). An expert pathol-ogist considered such superpixel images to retain the impor-tant structure of the tissue; tumor regions are still identifiableas are structures such as ducts and lobules.For each superpixel, we obtain a set of features to de-scribe its geometric and photometric properties as well as itsadjacency relationship to neighboring superpixels (Table 1).These features are normalized and concatenated to form a de-scriptor per superpixel.2.2. Bag-of-Superpixels (BoS)To perform superpixel classification, one approach is to usethe features directly. However this produces poor results dueto lack of context. To incorporate context, we use superpixelfeatures in the BoW framework.Extracted superpixel descriptors are quantized using a K-means dictionary to give visual words. A circular windowwith radius r is positioned at the centre point of the super-pixel to be classified. Visual words of superpixels withinthe circular window are histogrammed, resulting in a Bag-of-Superpixels (BoS). To provide complementary local features,the superpixel descriptor for the centre superpixel is concate-nated with BoS.2.3. Spatial Bag-of-Superpixels (S-BoS)Whilst BoS is a simple yet powerful representation, its maindrawback is lack of spatial information. Here, we model theInput: Image I , radius r, no. of rings Q, codebook C,no. of superpixels M ;Run SLIC on I to generate superpixels, s = [s1...sM ];Extract superpixel features, F = {f1...fM} ;for each superpixel, si, in s doIdentify superpixels t within circular window withradius, r, centered at si;Initialize spatial BoS histogram, Hi;for each superpixel, tj , in t doLookup codeword, vj for ftj in C;Compute d = ||c(tj)− c(si)||, c returns thecentre point of a superpixel;Increment Hi(vj ,⌊Qdr⌋);endNormalize Hi;endAlgorithm 1: Spatial Bag-of-Superpixels (S-BoS)spatial distribution of each visual word in a spatial Bag-of-Superpixel (S-BoS) histogram. Spatial information is cap-tured in the form of equally spaced rings within the circularwindow in BoS (Algorithm 1). This approach has some sim-ilarities to [8], however we analyze neighboring superpixelsfrom their visual words instead of RGB values. The resultinghistogram captures the BoS representation within Q rings.2.4. Rotation Invariant Superpixel PyramidSpatial pyramids partition an image repeatedly to computea BoW histogram per cell or sub-region. In [9], sub-regionsconsist of square grids and the representation is not rotationinvariant. Instead we use a Rotation Invariant SuperpixelPyramid (RISP) in which S-BoS histograms are computedper level (Fig. 2). The number of rings grows exponentiallywith p in each level. By utilizing circular rings, histogramscomputed from each ring are rotationally invariant. Previ-ous work [10] has used a technique similar to this for cellclassification. However, here we use superpixel visual wordsand describe a generalisable multi-level pyramid applicableto other computer vision tasks.S-BoS histograms are concatenated for each level to forma multiscale representation, RISP. We evaluate RISP and theS-BoS histogram, varying Q.2.5. Superpixel autocorrelogramIn previous work on scene categorization, spatial informationbetween interest regions has been modeled in the form of acodebook correlogram [11]. To improve computational costsby reducing the dimensionality of the correlogram, Huang etal [12] proposed the color autocorrelogram. We propose a su-perpixel autocorrelogram which captures spatial informationbetween pairs of superpixels. Similarly to [12], only identi-To appear in the Proceedings of 2015 IEEE International Symposium on Biomedical Imaging (ISBI 2015).Fig. 2. Levels 0, 1 and 2 of RISP. The BoW representation isas level 0 after which partitions are applied iteratively accord-ing to p. In the above, p = 2. Therefore for each ring in levell, two more are created in l + 1.cal pairs of codewords are counted. However we opt to usecircular windows instead of a regular grid to retain rotation in-variance. The distance distribution between superpixel centrepoints is captured resulting in a two-dimensional autocorrel-ogram. In reported experiments, 5 spatial bins were used.3. EVALUATIONA data set of 32 TMA tumor spots subjected to nuclear stain-ing for estrogen receptor (ER) was used. Spots were 0.6mm(3000 pixels) in diameter. Tumor regions were annotated bya highly experienced pathologist using a Bamboo tablet andstylus at 40x objective (Fig. 4).Tumor localization was evaluated using 8-fold cross-validation. A balanced training set was used with identicalnumber of positive and negative samples. A linear supportvector machine (SVM) classifier was implemented in the LI-BLINEAR framework. Grid search was performed to obtainan optimal cost parameter. 50, 000 SLIC superpixels wereextracted from each TMA spot image with the compactnessparameter set to 5. For a 3600x3600 pixel image, 50, 000SLIC superpixels can be computed in 29 seconds on an Inteli5-2410M 2.3GHz processor. A circular window with a ra-dius r = 100 pixels was used. TMA spots were segmentedmanually to reduce background interference.Figure 3 shows precision-recall curves for superpixel fea-tures with no context (SF), Bags-of-Superpixels (BoS), spatialBags-of-Superpixels (S-BoS), Rotation Invariant SuperpixelPyramids (RISP), superpixel autocorrelograms (Corr) and animplementation of a method described by Gorelick et al [8]which encapsulates RGB values in rings (Gorelick). Resultsare shown for 2 and 4 S-BoS rings and a 3-level RISP. Fea-tures from the central superpixel were concatenated to eachreported method to complement contextual information ex-tracted from the circular window. There is a clear improve-ment in classification performance between BoS and S-BoSwith 2 rings, showing the benefits of incorporating spatial in-formation. 3-level RISP which incorporates 7 (1+2+4) ringsalso shows a small improvement compared to S-BoS; how-0.3 0.4 0.5 0.6 0.7 0.8 0.90.30.40.50.60.70.80.9recallprecision  RISPS−BoS (Q=2)S−BoS (Q=4)BoSGorelickCorrSFFig. 3. Precision-recall curves for RISP, BoS, S-BoS, super-pixel features (SF), method as described in [8] (Gorelick) andthe superpixel autocorrelogram (Corr) with 200 codewords.Number of Codewords25 50 100 200BoS 0.623 0.644 0.670 0.676S-BoS Q = 2 0.638 0.657 0.680 0.688Q = 4 0.656 0.672 0.671 0.687RISP 2 levels {0, 1} 0.656 0.676 0.685 0.6903 levels {0, 1, 2} 0.645 0.676 0.686 0.692Table 2. F1 measures for BoS, S-BoS and RISP.ever increasing the number of rings in S-BoS results in minorimprovement. Both RISP and S-BoS show significant im-provement compared to the superpixel autocorrelogram andthe method described in [8].Table 2 shows F1 measures for RISP for various numbersof codewords. As the number of codewords increases andas more levels are incorporated in RISP, the accuracy contin-ues to increase; differences between 100 and 200 codewordsare marginal. Compared to BoS (i.e. level 0), RISP showsimprovement for all dictionary sizes. For comparison, ourprevious method [13] using pixel-level features including dif-ferential invariants up to 2nd order and intensity spin features,results in an F1 measure of 0.569 with a linear SVM. Com-pared to 3-level RISP this is a decrease of 0.123. Examplesof results achieved with RISP are shown in Fig. 4.A closer look at the difference images in Fig. 4 suggestsmost disagreements occur near the annotation boundary. Totest this theory, a distance transform was applied to binarymanual annotations. Tumor probabilities obtained using RISPwere thresholded at 0.5. The distribution of distances to theclosest boundary for pixels in disagreement with the pathol-ogist’s annotation is shown in Fig. 5. A large proportion ofdisagreements lies within a short distance of nearby tumorregions; 41% of disagreements are within 20 pixels (approxi-mately one cell diameter). This confirms most disagreementsbetween the pathologist’s and automated annotations occuraround the tumor boundary.To appear in the Proceedings of 2015 IEEE International Symposium on Biomedical Imaging (ISBI 2015).Image Annotation RISP DifferenceFig. 4. TMA images are shown on the left alongside man-ual annotations. The third column shows tumor probabilities(3-level RISP, 200 codewords). The last column contains dif-ference images where bright pixels are false positive and darkpixels are false negative errors.0 50 100 150 20000.050.10.150.2distance (pixels)proportion of pixels per spot  Disagreement pixelsAll pixelsFig. 5. Distribution of distances to the closest boundary forpixels within disagreement regions. The black line shows thedistribution of distances for all pixels.4. CONCLUSIONSuperpixel classification algorithms were proposed for theproblem of tumor localization in breast tissue. The bag-of-words framework was adapted to incorporate essentialspatial information. Results show spatial information in theform of circular rings improves classification performancecompared to Bags-of-Superpixels (BoS). When compared toGorelick [8], the proposed Rotational Invariant SuperpixelPyramid (RISP) performs favourably.Most disagreements between the pathologist and auto-mated segmentation occur near the tumor boundary. It isunlikely that hand-drawn manual annotations are accurate tothe pixel-level, therefore future work will consider how todeal with annotation uncertainty.5. REFERENCES[1] J. Kononen, L. Bubendorf, A. Kallionimeni, et al., “Tis-sue microarrays for high-throughput molecular profilingof tumor specimens,” Nature Medicine, vol. 4, no. 7, pp.844–847, 1998.[2] R. Achanta, A. Shaji, K. Smith, et al., “SLIC super-pixels compared to state-of-the-art superpixel methods,”PAMI, vol. 34, pp. 2274–2282, 2012.[3] Y. Al-Kofahi, W. Lassoued, W. Lee, and B. Roysam,“Improved automatic detection and segmentation of cellnuclei in histopathology images,” TBME, vol. 57, no. 4,pp. 841–852, 2010.[4] M.N. Gurcan, T. Pan, H. Shimada, and J. Saltz, “Imageanalysis for neuroblastoma classification: Segmentationof cell nuclei,” in EMBS, 2006, pp. 4844–4847.[5] C. Wang, “Robust automated tumour segmentation onhistological and immunohistochemical tissue images,”PLoS One, vol. 6, 2011.[6] A. Tabesh and M. Teverovskiy, “Tumor classification inhistological images of prostate using color texture,” inACSSC, 2006, pp. 841–845.[7] A.H. Beck, A.R. Sangoi, S.L. Leung, et al., “Systematicanalysis of breast cancer morphology uncovers stromalfeatures associated with survival,” Science TranslationalMedicine, vol. 3, 2011.[8] L. Gorelick, O. Veksler, M. Gaed, et al., “Prostatehistopathology: Learning tissue component histogramsfor cancer detection and classification,” TMI, vol. 32,pp. 1804–1818, 2013.[9] S. Lazebnik, C. Schmid, and J. Ponce, “Beyond bagsof features: Spatial pyramid matching for recognizingnatural scene categories,” in CVPR, 2006, vol. 2, pp.2169–2178.[10] A. Wiliem, C. Sanderson, Y. Wong, et al., “Automaticclassication of human epithelial type 2 cell indirect im-munouorescence images using cell pyramid matching,”Pattern Recognition, 2013.[11] Y. Zheng, H. Lu, C. Jin, and X. Xue, “Incorporatingspatial correlogram into bag-of-features model for scenecategorization,” ACCV 2009, pp. 333–342, 2010.[12] J. Huang, S.R. Kumar, M. Mitra, et al., “Image indexingusing color correlograms,” in CVPR, 1997, pp. 762–768.[13] S. J. McKenna, T. Amaral, S. Akbar, L. Jordan, andA. Thompson, “Immunohistochemical analysis ofbreast tissue microarray images using contextual clas-sifiers,” JPI, 2013.To appear in the Proceedings of 2015 IEEE International Symposium on Biomedical Imaging (ISBI 2015).

Tumor localization in tissue microarrays using rotation invariant superpixel pyramids

                                                              University of DundeeTumor localization in tissue microarrays using rotation invariant superpixel pyramidsAkbar, Shazia; Jordan, Lee; Thompson, Alastair M.; McKenna, StephenPublished in:2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI 2015)DOI:10.1109/ISBI.2015.7164111Publication date:2015Document VersionEarly version, also known as pre-printLink to publication in Discovery Research PortalCitation for published version (APA):Akbar, S., Jordan, L., Thompson, A. M., & McKenna, S. J. (2015). Tumor localization in tissue microarrays usingrotation invariant superpixel pyramids. In 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI2015). (pp. 1292-1295). IEEE. 10.1109/ISBI.2015.7164111General rightsCopyright and moral rights for the publications made accessible in Discovery Research Portal are retained by the authors and/or othercopyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated withthese rights. • Users may download and print one copy of any publication from Discovery Research Portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain. • You may freely distribute the URL identifying the publication in the public portal.Take down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Download date: 18. Mar. 2016TUMOR LOCALIZATION IN TISSUE MICROARRAYS USING ROTATION INVARIANTSUPERPIXEL PYRAMIDSShazia Akbar?, Lee Jordan†, Alastair M. Thompson‡, Stephen J. McKenna??School of Computing, University of Dundee, Dundee, UK†Pathology, Ninewells Hospital, Dundee, UK‡Department of Surgical Oncology, MD Anderson Cancer Center, Houston, United StatesABSTRACTTumor localization is an important component of histopathol-ogy image analysis; it has yet to be reliably automated forbreast cancer histopathology. This paper investigates the useof superpixel classification to localize tumor regions. A su-perpixel representation retains information about visual struc-tures such as cellular compartments, connective tissue, lumenand fatty tissue without having to commit to semantic seg-mentation at this level. In order to localize tumor in largeimages, a rotation invariant spatial pyramid representation isproposed using bags-of-superpixels. The method is evaluatedon expert-annotated oestrogen-receptor stained TMA spotsand compared to other superpixel classification techniques.Results demonstrate that it performs favorably.Index Terms— tumor classification, superpixels, spatialbag-of-words, rotation invariant spatial pyramid1. INTRODUCTIONTissue microarray (TMA) spots such as shown in Fig. 1(a) en-able relatively high-throughput inspection of various cancersfor multiple molecular markers [1]. They are used for val-idation of diagnostic markers in annotated clinical samples,testing new antibodies and probes, or determining optimalstaining conditions. The growth of tissue banks has lead toan increased workload for pathology experts, exceeding themanual skills available. Furthermore, expert pathology re-view suffers from inter- and intra-observer variations.Tumor localization is an important component of histo-pathology image analysis such as immunohistochemical(IHC) scoring and has not been reliably automated in breastTMAs. It is challenging because breast tissue has complexvisual structure; currently no detailed guidelines exist on howto visually identify tumor regions.One approach to automate tumor analysis is to first explic-itly detect and segment cells or subcellular components andthen build a feature representation based on these detectedobjects [3, 4]. However, in the images considered here, cel-lular compartment boundaries other than those of nuclei arenot often apparent. Nuclei detection is non-trivial because of(a) (b)Fig. 1. (a) A tissue microarray spot with annotated tumor re-gions. (b) An image patch (top) and a SLIC [2] superpixel im-age with each superpixel rendered as the average RGB valueof the pixels contained in it (bottom).the different cell types, varied appearance, and often heavilyclustered nuclei. An alternative approach is to treat the prob-lem as one of texture segmentation, without explicit detectionof objects [5,6]. However, statistical texture descriptors com-puted directly from the pixels may fail to capture structuralaspects of tissue.We investigate features based on superpixels to localizetumor regions. Fig. 1(b) shows a superpixel representation.The intuition is that a superpixel-based representation canretain information about visual structures such as cellularcompartments, connective tissue, lumen and fatty tissue with-out having to commit to semantic segmentation at this level.Superpixels have previously been applied in histopathologyfor cell nuclei segmentation [7] and tumor classification [8].Here, we use features from superpixels in a Bag-of-Words(BoW) framework, called Bags-of-Superpixels (BoS).We make the following contributions in this paper. (i) In-corporation of spatial information in BoS (iii) extension of thespatial pyramid [9] to build a Rotation Invariant SuperpixelPyramid (RISP) (iv) an adaptation of the autocorrelogram in-corporating superpixels. Tumor localization is formulated asTo appear in the Proceedings of 2015 IEEE International Symposium on Biomedical Imaging (ISBI 2015).Appearance Mean RGB, greyscale mean and variance,13 Haralick texture features (f1 − f13).Geometry Compactness, eccentricity, area, perimeter.Neighbors Number of immediate superpixel neigh-bors, variance of superpixel perimetershared with neighboring superpixels.Table 1. Features extracted from each superpixel.a superpixel classification problem and is evaluated on expert-annotated oestrogen-receptor stained TMA spots.2. METHODS2.1. Superpixel FeaturesA superpixel image representation is an over-segmentation inwhich similar pixels are grouped into perceptually consistentunits, regularized to be compact and of similar size. We useSLIC [2] to construct superpixels as it is computationally ef-ficient and retains tissue structure. The number of superpixelsis assigned such that the area of a single superpixel rarely ex-ceeds the area of a cell nucleus. Many cell nuclei are assignedtwo or more superpixels. An example SLIC representation ofa breast TMA patch is shown in Fig. 1(b) in which the aver-age superpixel area is 223 pixels (σ = 33). An expert pathol-ogist considered such superpixel images to retain the impor-tant structure of the tissue; tumor regions are still identifiableas are structures such as ducts and lobules.For each superpixel, we obtain a set of features to de-scribe its geometric and photometric properties as well as itsadjacency relationship to neighboring superpixels (Table 1).These features are normalized and concatenated to form a de-scriptor per superpixel.2.2. Bag-of-Superpixels (BoS)To perform superpixel classification, one approach is to usethe features directly. However this produces poor results dueto lack of context. To incorporate context, we use superpixelfeatures in the BoW framework.Extracted superpixel descriptors are quantized using a K-means dictionary to give visual words. A circular windowwith radius r is positioned at the centre point of the super-pixel to be classified. Visual words of superpixels withinthe circular window are histogrammed, resulting in a Bag-of-Superpixels (BoS). To provide complementary local features,the superpixel descriptor for the centre superpixel is concate-nated with BoS.2.3. Spatial Bag-of-Superpixels (S-BoS)Whilst BoS is a simple yet powerful representation, its maindrawback is lack of spatial information. Here, we model theInput: Image I , radius r, no. of rings Q, codebook C,no. of superpixels M ;Run SLIC on I to generate superpixels, s = [s1...sM ];Extract superpixel features, F = {f1...fM} ;for each superpixel, si, in s doIdentify superpixels t within circular window withradius, r, centered at si;Initialize spatial BoS histogram, Hi;for each superpixel, tj , in t doLookup codeword, vj for ftj in C;Compute d = ||c(tj)− c(si)||, c returns thecentre point of a superpixel;Increment Hi(vj ,⌊Qdr⌋);endNormalize Hi;endAlgorithm 1: Spatial Bag-of-Superpixels (S-BoS)spatial distribution of each visual word in a spatial Bag-of-Superpixel (S-BoS) histogram. Spatial information is cap-tured in the form of equally spaced rings within the circularwindow in BoS (Algorithm 1). This approach has some sim-ilarities to [8], however we analyze neighboring superpixelsfrom their visual words instead of RGB values. The resultinghistogram captures the BoS representation within Q rings.2.4. Rotation Invariant Superpixel PyramidSpatial pyramids partition an image repeatedly to computea BoW histogram per cell or sub-region. In [9], sub-regionsconsist of square grids and the representation is not rotationinvariant. Instead we use a Rotation Invariant SuperpixelPyramid (RISP) in which S-BoS histograms are computedper level (Fig. 2). The number of rings grows exponentiallywith p in each level. By utilizing circular rings, histogramscomputed from each ring are rotationally invariant. Previ-ous work [10] has used a technique similar to this for cellclassification. However, here we use superpixel visual wordsand describe a generalisable multi-level pyramid applicableto other computer vision tasks.S-BoS histograms are concatenated for each level to forma multiscale representation, RISP. We evaluate RISP and theS-BoS histogram, varying Q.2.5. Superpixel autocorrelogramIn previous work on scene categorization, spatial informationbetween interest regions has been modeled in the form of acodebook correlogram [11]. To improve computational costsby reducing the dimensionality of the correlogram, Huang etal [12] proposed the color autocorrelogram. We propose a su-perpixel autocorrelogram which captures spatial informationbetween pairs of superpixels. Similarly to [12], only identi-To appear in the Proceedings of 2015 IEEE International Symposium on Biomedical Imaging (ISBI 2015).Fig. 2. Levels 0, 1 and 2 of RISP. The BoW representation isas level 0 after which partitions are applied iteratively accord-ing to p. In the above, p = 2. Therefore for each ring in levell, two more are created in l + 1.cal pairs of codewords are counted. However we opt to usecircular windows instead of a regular grid to retain rotation in-variance. The distance distribution between superpixel centrepoints is captured resulting in a two-dimensional autocorrel-ogram. In reported experiments, 5 spatial bins were used.3. EVALUATIONA data set of 32 TMA tumor spots subjected to nuclear stain-ing for estrogen receptor (ER) was used. Spots were 0.6mm(3000 pixels) in diameter. Tumor regions were annotated bya highly experienced pathologist using a Bamboo tablet andstylus at 40x objective (Fig. 4).Tumor localization was evaluated using 8-fold cross-validation. A balanced training set was used with identicalnumber of positive and negative samples. A linear supportvector machine (SVM) classifier was implemented in the LI-BLINEAR framework. Grid search was performed to obtainan optimal cost parameter. 50, 000 SLIC superpixels wereextracted from each TMA spot image with the compactnessparameter set to 5. For a 3600x3600 pixel image, 50, 000SLIC superpixels can be computed in 29 seconds on an Inteli5-2410M 2.3GHz processor. A circular window with a ra-dius r = 100 pixels was used. TMA spots were segmentedmanually to reduce background interference.Figure 3 shows precision-recall curves for superpixel fea-tures with no context (SF), Bags-of-Superpixels (BoS), spatialBags-of-Superpixels (S-BoS), Rotation Invariant SuperpixelPyramids (RISP), superpixel autocorrelograms (Corr) and animplementation of a method described by Gorelick et al [8]which encapsulates RGB values in rings (Gorelick). Resultsare shown for 2 and 4 S-BoS rings and a 3-level RISP. Fea-tures from the central superpixel were concatenated to eachreported method to complement contextual information ex-tracted from the circular window. There is a clear improve-ment in classification performance between BoS and S-BoSwith 2 rings, showing the benefits of incorporating spatial in-formation. 3-level RISP which incorporates 7 (1+2+4) ringsalso shows a small improvement compared to S-BoS; how-0.3 0.4 0.5 0.6 0.7 0.8 0.90.30.40.50.60.70.80.9recallprecision  RISPS−BoS (Q=2)S−BoS (Q=4)BoSGorelickCorrSFFig. 3. Precision-recall curves for RISP, BoS, S-BoS, super-pixel features (SF), method as described in [8] (Gorelick) andthe superpixel autocorrelogram (Corr) with 200 codewords.Number of Codewords25 50 100 200BoS 0.623 0.644 0.670 0.676S-BoS Q = 2 0.638 0.657 0.680 0.688Q = 4 0.656 0.672 0.671 0.687RISP 2 levels {0, 1} 0.656 0.676 0.685 0.6903 levels {0, 1, 2} 0.645 0.676 0.686 0.692Table 2. F1 measures for BoS, S-BoS and RISP.ever increasing the number of rings in S-BoS results in minorimprovement. Both RISP and S-BoS show significant im-provement compared to the superpixel autocorrelogram andthe method described in [8].Table 2 shows F1 measures for RISP for various numbersof codewords. As the number of codewords increases andas more levels are incorporated in RISP, the accuracy contin-ues to increase; differences between 100 and 200 codewordsare marginal. Compared to BoS (i.e. level 0), RISP showsimprovement for all dictionary sizes. For comparison, ourprevious method [13] using pixel-level features including dif-ferential invariants up to 2nd order and intensity spin features,results in an F1 measure of 0.569 with a linear SVM. Com-pared to 3-level RISP this is a decrease of 0.123. Examplesof results achieved with RISP are shown in Fig. 4.A closer look at the difference images in Fig. 4 suggestsmost disagreements occur near the annotation boundary. Totest this theory, a distance transform was applied to binarymanual annotations. Tumor probabilities obtained using RISPwere thresholded at 0.5. The distribution of distances to theclosest boundary for pixels in disagreement with the pathol-ogist’s annotation is shown in Fig. 5. A large proportion ofdisagreements lies within a short distance of nearby tumorregions; 41% of disagreements are within 20 pixels (approxi-mately one cell diameter). This confirms most disagreementsbetween the pathologist’s and automated annotations occuraround the tumor boundary.To appear in the Proceedings of 2015 IEEE International Symposium on Biomedical Imaging (ISBI 2015).Image Annotation RISP DifferenceFig. 4. TMA images are shown on the left alongside man-ual annotations. The third column shows tumor probabilities(3-level RISP, 200 codewords). The last column contains dif-ference images where bright pixels are false positive and darkpixels are false negative errors.0 50 100 150 20000.050.10.150.2distance (pixels)proportion of pixels per spot  Disagreement pixelsAll pixelsFig. 5. Distribution of distances to the closest boundary forpixels within disagreement regions. The black line shows thedistribution of distances for all pixels.4. CONCLUSIONSuperpixel classification algorithms were proposed for theproblem of tumor localization in breast tissue. The bag-of-words framework was adapted to incorporate essentialspatial information. Results show spatial information in theform of circular rings improves classification performancecompared to Bags-of-Superpixels (BoS). When compared toGorelick [8], the proposed Rotational Invariant SuperpixelPyramid (RISP) performs favourably.Most disagreements between the pathologist and auto-mated segmentation occur near the tumor boundary. It isunlikely that hand-drawn manual annotations are accurate tothe pixel-level, therefore future work will consider how todeal with annotation uncertainty.5. REFERENCES[1] J. Kononen, L. Bubendorf, A. Kallionimeni, et al., “Tis-sue microarrays for high-throughput molecular profilingof tumor specimens,” Nature Medicine, vol. 4, no. 7, pp.844–847, 1998.[2] R. Achanta, A. Shaji, K. Smith, et al., “SLIC super-pixels compared to state-of-the-art superpixel methods,”PAMI, vol. 34, pp. 2274–2282, 2012.[3] Y. Al-Kofahi, W. Lassoued, W. Lee, and B. Roysam,“Improved automatic detection and segmentation of cellnuclei in histopathology images,” TBME, vol. 57, no. 4,pp. 841–852, 2010.[4] M.N. Gurcan, T. Pan, H. Shimada, and J. Saltz, “Imageanalysis for neuroblastoma classification: Segmentationof cell nuclei,” in EMBS, 2006, pp. 4844–4847.[5] C. Wang, “Robust automated tumour segmentation onhistological and immunohistochemical tissue images,”PLoS One, vol. 6, 2011.[6] A. Tabesh and M. Teverovskiy, “Tumor classification inhistological images of prostate using color texture,” inACSSC, 2006, pp. 841–845.[7] A.H. Beck, A.R. Sangoi, S.L. Leung, et al., “Systematicanalysis of breast cancer morphology uncovers stromalfeatures associated with survival,” Science TranslationalMedicine, vol. 3, 2011.[8] L. Gorelick, O. Veksler, M. Gaed, et al., “Prostatehistopathology: Learning tissue component histogramsfor cancer detection and classification,” TMI, vol. 32,pp. 1804–1818, 2013.[9] S. Lazebnik, C. Schmid, and J. Ponce, “Beyond bagsof features: Spatial pyramid matching for recognizingnatural scene categories,” in CVPR, 2006, vol. 2, pp.2169–2178.[10] A. Wiliem, C. Sanderson, Y. Wong, et al., “Automaticclassication of human epithelial type 2 cell indirect im-munouorescence images using cell pyramid matching,”Pattern Recognition, 2013.[11] Y. Zheng, H. Lu, C. Jin, and X. Xue, “Incorporatingspatial correlogram into bag-of-features model for scenecategorization,” ACCV 2009, pp. 333–342, 2010.[12] J. Huang, S.R. Kumar, M. Mitra, et al., “Image indexingusing color correlograms,” in CVPR, 1997, pp. 762–768.[13] S. J. McKenna, T. Amaral, S. Akbar, L. Jordan, andA. Thompson, “Immunohistochemical analysis ofbreast tissue microarray images using contextual clas-sifiers,” JPI, 2013.To appear in the Proceedings of 2015 IEEE International Symposium on Biomedical Imaging (ISBI 2015).

Automatic classication of human epithelial type 2 cell indirect immunouorescence images using cell pyramid matching,” Pattern Recognition,

Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories,” in CVPR,

Image analysis for neuroblastoma classification: Segmentation of cell nuclei,”

Image indexing using color correlograms,” in CVPR,

Immunohistochemical analysis of breast tissue microarray images using contextual classifiers,” JPI,

Improved automatic detection and segmentation of cell nuclei in histopathology images,”

Incorporating spatial correlogram into bag-of-features model for scene categorization,” ACCV

Prostate histopathology: Learning tissue component histograms for cancer detection and classification,”

Robust automated tumour segmentation on histological and immunohistochemical tissue images,”

SLIC superpixels compared to state-of-the-art superpixel methods,”

Systematic analysis of breast cancer morphology uncovers stromal features associated with survival,”

Tissue microarrays for high-throughput molecular profiling of tumor specimens,”

Tumor classification in histological images of prostate using color texture,” in ACSSC,

Tumor localization in tissue microarrays using rotation invariant superpixel pyramids

Abstract

Similar works

Full text

Available Versions

University of Dundee Online Publications

University of Dundee Online Publications

Crossref