Three dimensional (3D) tissue reconstructions from the histology images with different stains allows the spatial alignment of structural and functional elements highlighted by different stains for quantitative study of many physiological and pathological phenomena. This has significant potential to improve the understanding of the growth patterns and the spatial arrangement of diseased cells, and enhance the study of biomechanical behavior of the tissue structures towards better treatments (e.g. tissue-engineering applications)

Bulpitt, AJ

Magee, DR

Song, Y

Treanor, D

Journal of Pathology Informatics

PubMed

White Rose Research Online

promoting access to White Rose research papersWhite Rose Research Onlineeprints@whiterose.ac.ukUniversities of Leeds, Sheffield and Yorkhttp://eprints.whiterose.ac.uk/This is the published version of an article in the Journal of PathologyInformatics, 4 (Supple)White Rose Research Online URL for this paper:http://eprints.whiterose.ac.uk/id/eprint/77413Published article:Song, Y, Treanor, D, Bulpitt, AJ and Magee, DR (2013) 3D reconstruction ofmultiple stained histology images. Journal of Pathology Informatics, 4 (Supple). 1- 5. ISSN 2229-5089http://dx.doi.org/10.4103/2153-3539.109864J Pathol Inform Editor-in-Chief:Anil V. Parwani , Liron Pantanowitz,Pittsburgh, PA, USA Pittsburgh, PA, USAFor entire Editorial Board visit : www.jpathinformatics.org/editorialboard.aspOPEN ACCESSHTML formatSymposium - Original Research3D reconstruction of multiple stained histology imagesYi Song, Darren Treanor, Andrew J. Bulpitt, Derek R. MageeSchool of Computing, University of Leeds, Leeds, UKE-mail: *Yi Song - y.song@leeds.ac.uk*Corresponding authorReceived: 21 January 13 Accepted: 21 January 13 Published: 30 March 13AbstractContext:Three dimensional (3D) tissue reconstructions from the histology images withdifferent stains allows the spatial alignment of structural and functional elements highlightedby different stains for quantitative study of many physiological and pathological phenomena.and the spatial arrangement of diseased cells, and enhance the study of biomechanicalbehavior of the tissue structures towards better treatments (e.g. tissue-engineeringapplications).Methods:This paper evaluates three strategies for 3D reconstruction fromsets of two dimensional (2D) histological sections with different stains, by combiningmethods of 2D multi-stain registration and 3D volumetric reconstruction from samestain sections. Setting and Design:The different strategies have been evaluated on twoliver specimens (80 sections in total) stained with Hematoxylin and Eosin (H and E),SiriusRed, and Cytokeratin (CK) 7. Results and Conclusion: A strategy of using multi-stainregistration to align images of a second stain to a volume reconstructed by same-stainregistration results in the lowest overall error, although an interlaced image registrationapproach may be more robust to poor section quality.Key words: 3D reconstruction, computerized diagnosis, digital pathology, histologicalimage, image analysis, multi-stain registrationCopyright: © 2013 SongY.This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, andreproduction in any medium, provided the original author and source are credited.This article may be cited as:SongY,Treanor D, Bulpitt AJ, Magee DR. 3D reconstruction of multiple stained histology images. J Pathol Inform 2013;4:7.Available FREE in open access from: http://www.jpathinformatics.org/text.asp?2013/4/2/7/109864Access this article onlineWebsite:www.jpathinformatics.orgDOI: 10.4103/2153-3539.109864Quick Response Code:INTRODUCTIONHistopathology is the microscopic examination of tissuesections, which are colored with different stains todisplay different functional or physical properties. Threedimensional (3D) tissue reconstruction from the digitalimages of serial sections has significant potential toimprove the understanding of the growth patterns and thespatial arrangement of diseased cells, enhance the studyof biomechanical behavior of the tissue structures towardsbetter treatments (e.g. tissue-engineering applications). Thiscan be obtained by successively applying two dimensional(2D) image-to-image registrations and then concatenatingthe set of aligned images to form a 3D volumetric dataset.Various methods based on this idea have been presented for3D reconstruction from a set of images of same-stain serialsections.[1-4] Recently, there is increasing research interestin registering 2D histology images of consecutive sectionswith different stains,[5-8], since automatic co-registration ofthese images allows the spatial alignment of structural andfunctional elements in different modalities for quantitativestudy of many physiological and pathological phenomena.However, there is no work known to the authors addressingthe problem of 3D reconstruction from images of sectionswith different stains. In this paper, we discuss threestrategies for combining the solutions of automatic 2Dmulti-stain registration and 3D volumetric reconstructionfrom same-stain sections. As the 2D multi-stain registrationJ Pathol Inform 2013, 1:7 http://www.jpathinformatics.org/content/4/1/7and same-stain volumetric reconstruction are the key stepsof all 3D reconstruction strategies, for the completeness, weprovide the necessary details about our implementations of2D registrations in section Methods.MATERIALSTwo human liver surgical Specimens (A, B) are used toexamine three 3D reconstruction strategies. Local NHSresearch ethical approval was obtained for use of the humantissue. On Specimen A, four artificial vertical holes weremade for the quantitative evaluation of the accuracy of the3D reconstruction. Specimen B was chosen because thereare relatively consistent anatomic features present in thestack of sections. Using standard histological techniques, thespecimens were formalin fixed, paraffin embedded, and cutinto ~5µm sections using a microtome.For Specimen A, every 4th section was selected andstained with H and E; every 5th section was stainedwith Sirius Red. Twenty five sections of each stainwere prepared, with 50 sections in total.For Specimen B, every 2nd, 3rd, 4th section was stainedwith H and E while every 6th, 7th, 8th sections werestained with CK7. Fifteen sections of each stain wereprepared, with 30 sections in total.Sections were scanned at 20× magnification with averageimage size ~1.4 G (Specimen A) and 40× with averageimage size ~18.5 G (specimen B) using an Aperio XTscanner. To verify the influence of misaligned 2D imageson the various 3D reconstruction strategies, images ofsections with the following problems are included:Luminance gradient: Sections mounted close tothe edge of the glass slide produce images withsignificant luminance gradients [Figure 1-A1, 1-A2].Non-tissue noise: Dust and air bubbles in the slide[Figure 1-A1, 1-A2].Damaged and missing sections: During sectioningand mounting, sections are occasionally torn, folded[Figure 1-B1].Staining variations: Differences in section thickness,staining duration, and stain concentration result incolour variations [Figure 1-B2].METHODSThere are several potential approaches for 3D reconstructionfrom images of differently stained sections which we haveimplemented and evaluated. An intuitive option is to apply2D registration sequentially for each stain separately (i.e.same-stain registration) after the reference image (i.e. thefirst section) for each stack has been aligned by applying2D multi-stain registration Figure 2, Strategy 1). Thesecond solution is to apply same-stain registration on thestack of images with one stain (e.g. T1) sequentially andthen aligning every image in the stack of the other stain(e.g. T2) to its adjacent image in the stack of stain T1using multi-stain registration [Figure 2, Strategy 2]. Thethird approach is interlacing images of stain T1 and T2in the same sequence as sectioning and applying multi-stain registration for all neighboring images [Figure 2,Strategy 3]. In “Results” section, we present a comparisonstudy of these options. The 3D registration approachesdiscussed in this paper are the extension of: 1) Our previousworks on 3D volume reconstruction from 2D registrationof images of same-stain serial sections[1] and 2) ourunsupervised content classification based on 2D multi-stainregistration method (manuscript under preparation). Tomake this paper independently readable, we briefly reviewour 2D registration methods in the following sections.Automatic 2D:2D Same-Stain Registration ofHistology ImagesRegistration consists in two main steps: 1) Initial rigidalignment of an red, green, blue (RGB) image pair at lowresolution X0 using a phase correlation based method (acombination of [9] and [10] to recover scale and rotation)Figure 1: Example histological sections on which the green marks are annotations used for the evaluationFigure 2: Strategies of 3D registration of multi-stain pathologicalimagesJ Pathol Inform 2013, 1:7 http://www.jpathinformatics.org/content/4/1/7applied to grayscale image; 2) Non-rigid alignment isdone by dividing each image into small patches (fromour cross-validation study, the best results are given at256 × 256 pixels per patch) and computing the rigidtransformation of these patches separately by applying thephase correlation based method. The deformation field isthen represented by five vectors for each patch, i.e. oneat each corner and one at the centre. A cubic B-Splinetransform is estimated to approximate the deformationfield using regularized least squares error minimization.[11]The non-rigid transform in Step 2 is refined by a multi-resolution (from coarse X1 to fine Xn) approach. Thepatch matching approach is the key to tackle large imagesize (gigabytes at high resolution).Automatic 2D:2D Multi-Stain Registration ofHistology ImagesA challenge when registering histology images withdifferent stains is that different stains highlight differentsubstances in tissues, resulting in dissimilar structuralappearances on adjacent tissue sections. We addressthis challenge by proposing an unsupervised contentclassification method, which automatically identifiescommon content classes from differently stainedhistology image pair. This method creates two multi-channel probability images [Figure 3c] for the non-rigidalignment, as in step 2 where the grayscale images areused for the same-stain registration. The workflow ofmulti-stain registration is illustrated in Figure 3, andthe results at resolution Xi i n) of each stepare exemplified on the right hand side. The contentclassification algorithm is summarized in Table 1, wherewe assume the input image pair at resolution Xi hasbeen pre-registered by the transformation estimated atresolution Xi-1 (within a multi-resolution framework).Spatial location features are introduced as complementaryto the appearance features. This overcomes the problemwhere the appearance features alone (for some stains) arenot sufficient to separate certain image regions, whichhave similar appearance, but belong to different contentclasses. As that the content classes are emergent fromco-occurrence statistics between the image pairs, they donot necessarily resemble to real anatomic classes. Notethis is not a prerequisite for registration.Input: RGB color image pair, i.e., reference image IRimage IF.Output:A pair of multi-channel probability images.1. Create appearance feature vector per pixel uR (x, y),uF (x, y), which consists of feature vectors set uR {uR (x, y)},uF {uF (x, y)}2. Cluster feature vector sets UR, UF into NR, NF clusters,respectively, obtaining cluster-label images , .3. Find the best partition functions , which partitionNR, NF clusters into T content classes, by maximizing MutualInformation .features till the convergance condition is met (a threshold onthe partition error).5. Convert the original color image pair into a T-channelprobability image pair.registration of multi-stain histology images. Examples results ofeach step are illustrated at the right side.(a) Original tissue sectionsstained with H and E and Sirius Red stains.(b) Rigidly aligned imagepair. (c) Example output of three channel probability image pair.Each channel is corresponding to a content class automaticallydcbaFigure 4: Boxplot of the alignment accuracy. x-axis indicates theannotations. y-axis is the Hausdorff distance (µm).The solid linesmethod.The dash lines indicate the results without using contenta bcJ Pathol Inform 2013, 1:7 http://www.jpathinformatics.org/content/4/1/7RESULTSEvaluation on 2D Multi-Stain RegistrationThe 2D unsupervised content classification-basedregistration method plays a key role in our 3Dreconstruction study. Thus, we evaluate the 2D multi-stain registration method using three histological imagepairs with six different stains from a liver specimen, aintervertebral disc specimen, and a colon specimen. Wemanually annotate 20~30 regions (the number of pixelsper annotation ranges 2000~5600) on each image pair.The alignment accuracy for each image pair is evaluatedby measuring Hausdorff distance (µm) of correspondingannotations at full-resolution level (at which the imagewas originally scanned). The boxplot of registrationaccuracy is shown in Figure 4.Evaluation of 3D Reconstruction StrategiesFigure 5 illustrates the influence of different 3Dreconstruction strategies (described in “Materials andMethods” section) on the reconstructed specimens bystacking all the registered images of stain A serially aswhen they were cut. For example, for the reconstructedSpecimen A, the odd number images are those stainedwith H and E and the even number images are stainedwith Sirius Red. The Hausdorff distances (µm) aremeasured at 20× magnification for Specimen A. Thegraphs at the position n depict the distance between then+1th images and the 1st image (i.e. accumulated error).Each line plotted in the graph corresponds to one manualannotation, as shown in Figure 1.Figure 5d (Strategy 1) shows large misalignments existbetween successive images of different stains, due to thefact that Strategy1 is based on two sets of independentsame-stain registrations. This results in a large accumulatederror with respect to the first image [Figure 5a]. Strategy 2shows better registration accuracy between stains [Figure5d] resulted from applying same-stain registration on oddnumber images (H and E stain) followed by multi-stainregistration to align every even number images (Sirius Redstain) to its neighboring H and E image. There is alsoless accumulation of errors with respect to the first image[Figure 5b]. Figure 5d illustrates the Strategy 3 also hasgood registration accuracy between successive images ofdifferent stains; however, accumulated error is larger thanStrategy 2 [Figure 5c].Unlike Specimen A, there are no artificial markers on theSpecimen B. As the influence of anatomic changes overridesthe registration errors, for Specimen B, we do not provideHausdorff evaluation based on annotations of real anatomicstructures. Instead, the smoothness of the reconstructedspecimens in coronal views offers a more intuitive qualitativeway of comparing the three strategies, as in Figure 6.Figure 6 shows a coronal view of the reconstructed specimens.number of sections. (d) Boxplot of the Hausdorff distances between successive images in each stackdcbaJ Pathol Inform 2013, 1:7 http://www.jpathinformatics.org/content/4/1/7For Specimen A (top), Strategy 2 gives the smoothest 3Dreconstruction, however, several outliers exist. Strategy 1 hasa smooth reconstruction before the first big misalignmentpresents, which leads to the poor reconstruction of therest part of the specimen. Although Strategy 3 has thesmallest Hausdorff distance between successive images, asillustrated in Figure 5d, the reconstructed volume is notas smooth as Strategy 2, reflecting the fact that Strategy 3has random accumulated registration errors. For SpecimenB (bottom), similar conclusions can be drawn. Strategy 1has larger registration error at the end of the reconstructedstack than other two strategies. Strategy 2 and 3 give muchsmoother blood vessel walls and specimen boundary, andtheir difference is minor in this case.In addition to the registration accuracy, computationalefficiency is also an important factor to be considered.Table 2 exemplifies the execution times of same-stainand multi-stain 2D registrations at different resolutions.All computations were carried out on a desktop computerwith the following specifications: Intel i7 dual core×3.07 GHz, 8 GB RAM, Windows 7. The code is not yetoptimized for parallel computation.CONCLUSIONIn our experiments, we also notice that the multi-stain2D registration method is less sensitive to problematicsections and performs accurately on serial sections ofcontinuous cut. However, when the gap between the twosections increases (i.e. more sections are left out of thereconstruction), the registration accuracy correspondinglydecreases since larger variations in tissue structures arepresent between the two sections, and thus affects theFigure 6:Comparison of 3D volumetric reconstructions of SpecimenCoronal views of the reconstructed Specimen B (Bottom).Viewplane locates at annotations b2result of co-occurrence statistics, and, thus, unsupervisedclassification performance. In summary, Strategy 3 (onlyinvolving multi-stain 2D registration) has an advantage indealing with some problematic sections and provides good3D reconstruction results (when only few sections in thecut are left out of 3D reconstruction), however, it requires2×n-1 multi-stain registrations (n the number of images ineach stain). Strategy 2 results in the lowest accumulatederrors if those problematic sections can be discarded beforethe registration, but is more sensitive to these problematicsections. Moreover, it only requires n-1 same-stain 2Dregistrations plus n multi-stain 2D registrations, whichis much more computationally efficient than Strategy 3.Both Strategy 2 and Strategy 3 are practically usable.REFERENCES1. Roberts N, Magee D, SongY, Brabazon K, Shires M, Crellin D, et al.Towardsroutine use of 3D histopathology as a research tool. American Journal ofPathology, 2012;180:1835-1842.2. Kurien T, Boyce RW, Paish EC, Ronan J, Maddison J, Rakha EA, et al. Threedimensional reconstruction of a human breast carcinoma using routinelaboratory equipment and immunohistochemistry. Journal of ClinicalPathology 2005;58:968-972.3. Magee D, Treanor D, Quirke P. A new image registration algorithm withapplication to 3D histopathology. Proceedings of the 3rd Medical ImageComputing and Computer Assisted Intervention (MICCAI) Workshop onMicroscopic Image Analysis with Applications in Biology (MIAAB). NewYork; September 5-6, 2008.4.of biological images for volume reconstruction. Medical Image Analysis2006;10:465-483.5. Can A, Bello M, Cline HM,Tao X, Ginty F, Sood A, et al. Multi-modal imagingof histological tissue sections. Biomedical Imaging: From Nano to Macro,2008. ISBI 2008. 5th IEEE International Symposium. Paris; May 14-17, 2008.6. Cooper L, Sertel O, Kong J, Lozanski G, Huang K, Gurcan M. Feature-basedregistration of histopathology images with different stains: An applicationfor computerized follicular lymphoma prognosis. Computer MethodsPrograms in Biomedicine. 2009;96:182-92.7. Cooper L, Naidu S, Leone G, Saltzl J, Huang K. Registering high resolutionmicroscopic images with different histochemical staining - a tool formapping gene expression with cellular structures. Proceedings of the 3rdMedical Image Computing and Computer Assisted Intervention (MICCAI)Workshop on Microscopic Image Analysis with Applications in Biology(MIAAB). Piscataway; September 21, 2007.8. Mueller D,Vossen D, Hulsken B. Real-time deformable registration of multi-modal whole slides for digital pathology. Computerized Medical Imaging andGraphics. 2011;35:542-556.9. Casasent D, Psaltis D. Position, rotation, and scale invariant opticalcorrelation.Applied Optics 1976;15:1795-1799.10. Kuglin C,Hines D.The phase correlation image alignment method.Proceedingsof IEEE International Conference Cybernetics and Society; 1975. p. 163-165.11.linear inversion. Society for Industrial and Applied Mathematics (SIAM) 1999.Resolutions Specimen A Specimen BSame-stainTimesMulti-stainTimesSame-stainTimesMulti-stainTimes×1.25 1400 1360 16s 87s 3000×2200 44s 242s×2.5 2800 2720 56s 240s 6000×4400 176s 936s

3D reconstruction of multiple stained histology images.

Yi Song

Darren Treanor

Andrew J. Bulpitt

Derek R. Magee

Crossref

3D reconstruction of multiple stained histology images

Context: Three dimensional (3D) tissue reconstructions from the histology images with different stains allows the spatial alignment of structural and functional elements highlighted by different stains for quantitative study of many physiological and pathological phenomena. This has significant potential to improve the understanding of the growth patterns and the spatial arrangement of diseased cells, and enhance the study of biomechanical behavior of the tissue structures towards better treatments (e.g. tissue-engineering applications). Methods: This paper evaluates three strategies for 3D reconstruction from sets of two dimensional (2D) histological sections with different stains, by combining methods of 2D multi-stain registration and 3D volumetric reconstruction from same stain sections. Setting and Design: The different strategies have been evaluated on two liver specimens (80 sections in total) stained with Hematoxylin and Eosin (H and E), Sirius Red, and Cytokeratin (CK) 7. Results and Conclusion: A strategy of using multi-stain registration to align images of a second stain to a volume reconstructed by same-stain registration results in the lowest overall error, although an interlaced image registration approach may be more robust to poor section quality

Andrew J Bulpitt

Derek R Magee

Directory of Open Access Journals

3D reconstruction of multiple stained histology images.

Abstract

Similar works

Full text

Available Versions

White Rose Research Online

Crossref

Directory of Open Access Journals