3D fusion of histology to multi-parametric MRI for prostate cancer imaging evaluation and lesion-targeted treatment planning

Multi-parametric magnetic resonance imaging (mpMRI) of localized prostate cancer has the potential to support detection, staging and localization of tumors, as well as selection, delivery and monitoring of treatments. Delineating prostate cancer tumors on imaging could potentially further support the clinical workflow by enabling precise monitoring of tumor burden in active-surveillance patients, optimized targeting of image-guided biopsies, and targeted delivery of treatments to decrease morbidity and improve outcomes. Evaluating the performance of mpMRI for prostate cancer imaging and delineation ideally includes comparison to an accurately registered reference standard, such as prostatectomy histology, for the locations of tumor boundaries on mpMRI. There are key gaps in knowledge regarding how to accurately register histological reference standards to imaging, and consequently further gaps in knowledge regarding the suitability of mpMRI for tasks, such as tumor delineation, that require such reference standards for evaluation. To obtain an understanding of the magnitude of the mpMRI-histology registration problem, we quantified the position, orientation and deformation of whole-mount histology sections relative to the formalin-fixed tissue slices from which they were cut. We found that (1) modeling isotropic scaling accounted for the majority of the deformation with a further small but statistically significant improvement from modeling affine transformation, and (2) due to the depth (mean±standard deviation (SD) 1.1±0.4 mm) and orientation (mean±SD 1.5±0.9°) of the sectioning, the assumption that histology sections are cut from the front faces of tissue slices, common in previous approaches, introduced a mean error of 0.7 mm. To determine the potential consequences of seemingly small registration errors such as described above, we investigated the impact of registration accuracy on the statistical power of imaging validation studies using a co-registered spatial reference standard (e.g. histology images) by deriving novel statistical power formulae that incorporate registration error. We illustrated, through a case study modeled on a prostate cancer imaging trial at our centre, that submillimeter differences in registration error can have a substantial impact on the required sample sizes (and therefore also the study cost) for studies aiming to detect mpMRI signal differences due to 0.5 – 2.0 cm3 prostate tumors. With the aim of achieving highly accurate mpMRI-histology registrations without disrupting the clinical pathology workflow, we developed a three-stage method for accurately registering 2D whole-mount histology images to pre-prostatectomy mpMRI that allowed flexible placement of cuts during slicing for pathology and avoided the assumption that histology sections are cut from the front faces of tissue slices. The method comprised a 3D reconstruction of histology images, followed by 3D–3D ex vivo–in vivo and in vivo–in vivo image transformations. The 3D reconstruction method minimized fiducial registration error between cross-sections of non-disruptive histology- and ex-vivo-MRI-visible strand-shaped fiducials to reconstruct histology images into the coordinate system of an ex vivo MR image. We quantified the mean±standard deviation target registration error of the reconstruction to be 0.7±0.4 mm, based on the post-reconstruction misalignment of intrinsic landmark pairs. We also compared our fiducial-based reconstruction to an alternative reconstruction based on mutual-information-based registration, an established method for multi-modality registration. We found that the mean target registration error for the fiducial-based method (0.7 mm) was lower than that for the mutual-information-based method (1.2 mm), and that the mutual-information-based method was less robust to initialization error due to multiple sources of error, including the optimizer and the mutual information similarity metric. The second stage of the histology–mpMRI registration used interactively defined 3D–3D deformable thin-plate-spline transformations to align ex vivo to in vivo MR images to compensate for deformation due to endorectal MR coil positioning, surgical resection and formalin fixation. The third stage used interactively defined 3D–3D rigid or thin-plate-spline transformations to co-register in vivo mpMRI images to compensate for patient motion and image distortion. The combined mean registration error of the histology–mpMRI registration was quantified to be 2 mm using manually identified intrinsic landmark pairs. Our data set, comprising mpMRI, target volumes contoured by four observers and co-registered contoured and graded histology images, was used to quantify the positive predictive values and variability of observer scoring of lesions following the Prostate Imaging Reporting and Data System (PI-RADS) guidelines, the variability of target volume contouring, and appropriate expansion margins from target volumes to achieve coverage of histologically defined cancer. The analysis of lesion scoring showed that a PI-RADS overall cancer likelihood of 5, denoting “highly likely cancer”, had a positive predictive value of 85% for Gleason 7 cancer (and 93% for lesions with volumes \u3e0.5 cm3 measured on mpMRI) and that PI-RADS scores were positively correlated with histological grade (ρ=0.6). However, the analysis also showed interobserver differences in PI-RADS score of 0.6 to 1.2 (on a 5-point scale) and an agreement kappa value of only 0.30. The analysis of target volume contouring showed that target volume contours with suitable margins can achieve near-complete histological coverage for detected lesions, despite the presence of high interobserver spatial variability in target volumes. Prostate cancer imaging and delineation have the potential to support multiple stages in the management of localized prostate cancer. Targeted biopsy procedures with optimized targeting based on tumor delineation may help distinguish patients who need treatment from those who need active surveillance. Ongoing monitoring of tumor burden based on delineation in patients undergoing active surveillance may help identify those who need to progress to therapy early while the cancer is still curable. Preferentially targeting therapies at delineated target volumes may lower the morbidity associated with aggressive cancer treatment and improve outcomes in low-intermediate-risk patients. Measurements of the accuracy and variability of lesion scoring and target volume contouring on mpMRI will clarify its value in supporting these roles

Intensity Based Non-rigid Registration of 3D Whole Mouse Optical and MR Image Volumes

Novel magnetic resonance (MR) imaging techniques can be validated using accurate co-registration with histology. Whole-animal histological sections allow for simultaneous analysis of multiple tissues, and may also aid in registration by providing contextual information and structural support to tissues which if isolated from the body would be difficult to register. This thesis explores the feasibility of co-registration between whole mouse histology with 3D MR images using an intermediate optical image volume acquired during tissue sectioning. Of the two transformations required for this approach, 3D co-registration of MR and optical images is more challenging to perform due to changes in contrast, slice orientation, and resolution between these modalities. Here, an automated non-rigid registration technique utilizing mutual information is proposed to accurately register 3D whole mouse optical and MR images as a first step towards automated registration of histology. Validation of this technique was accomplished through calculation of post-registration target registration error

A method for accurate spatial registration of PET images and histopathology slices

Background: Accurate alignment between histopathology slices and positron emission tomography (PET) images is important for radiopharmaceutical validation studies. Limited data is available on the registration accuracy that can be achieved between PET and histopathology slices acquired under routine pathology conditions where slices may be non-parallel, non-contiguously cut and of standard block size. The purpose of this study was to demonstrate a method for aligning PET images and histopathology slices acquired from patients with laryngeal cancer and to assess the registration accuracy obtained under these conditions. Methods: Six subjects with laryngeal cancer underwent a 64Cu-copper-II-diacetyl-bis(N4-methylthiosemicarbazone) (64Cu-ATSM) PET computed tomography (CT) scan prior to total laryngectomy. Sea urchin spines were inserted into the pathology specimen to act as fiducial markers. The specimen was fixed in formalin, as per standard histopathology operating procedures, and was then CT scanned and cut into millimetre-thick tissue slices. A subset of the tissue slices that included both tumour and fiducial markers was taken and embedded in paraffin blocks. Subsequently, microtome sectioning and haematoxylin and eosin staining were performed to produce 5-μm-thick tissue sections for microscopic digitisation. A series of rigid registration procedures was performed between the different imaging modalities (PET; in vivo CT—i.e. the CT component of the PET-CT; ex vivo CT; histology slices) with the ex vivo CT serving as the reference image. In vivo and ex vivo CTs were registered using landmark-based registration. Histopathology and ex vivo CT images were aligned using the sea urchin spines with additional anatomical landmarks where available. Registration errors were estimated using a leave-one-out strategy for in vivo to ex vivo CT and were estimated from the RMS landmark accuracy for histopathology to ex vivo CT. Results: The mean ± SD accuracy for registration of the in vivo to ex vivo CT images was 2.66 ± 0.66 mm, and the accuracy for registration of histopathology to ex vivo CT was 0.86 ± 0.41 mm. Estimating the PET to in vivo CT registration accuracy to equal the PET-CT alignment accuracy of 1 mm resulted in an overall average registration error between PET and histopathology slices of 3.0 ± 0.7 mm. Conclusions: We have developed a registration method to align PET images and histopathology slices with an accuracy comparable to the spatial resolution of the PET images.</p

A Survey on Deep Learning in Medical Image Analysis

Deep learning algorithms, in particular convolutional networks, have rapidly become a methodology of choice for analyzing medical images. This paper reviews the major deep learning concepts pertinent to medical image analysis and summarizes over 300 contributions to the field, most of which appeared in the last year. We survey the use of deep learning for image classification, object detection, segmentation, registration, and other tasks and provide concise overviews of studies per application area. Open challenges and directions for future research are discussed.Comment: Revised survey includes expanded discussion section and reworked introductory section on common deep architectures. Added missed papers from before Feb 1st 201

Registration of in-vivo to ex-vivo MRI of surgically resected specimens: A pipeline for histology to in-vivo registration.

BACKGROUND: Advances in MRI have the potential to improve surgical treatment of epilepsy through improved identification and delineation of lesions. However, validation is currently needed to investigate histopathological correlates of these new imaging techniques. The purpose of this work is to develop and evaluate a protocol for deformable image registration of in-vivo to ex-vivo resected brain specimen MRI. This protocol, in conjunction with our previous work on ex-vivo to histology registration, completes a registration pipeline for histology to in-vivo MRI, enabling voxel-based validation of novel and existing MRI techniques with histopathology. NEW METHOD: A combination of image-based and landmark-based 3D registration was used to register in-vivo MRI and the ex-vivo MRI from patients (N=10) undergoing epilepsy surgery. Target registration error (TRE) was used to assess accuracy and the added benefit of deformable registration. RESULTS: A mean TRE of 1.35±0.11 and 1.41±0.33mm was found for neocortical and hippocampal specimens respectively. Statistical analysis confirmed that the deformable registration significantly improved the registration accuracy for both specimens. COMPARISON WITH EXISTING METHODS: Image registration of surgically resected brain specimens is a unique application which presents numerous technical challenges and that have not been fully addressed in previous literature. Our computed TRE are comparable to previous attempts tackling similar applications, as registering in-vivo MRI to whole brain or serial histology. CONCLUSION: The presented registration pipeline finds dense and accurate spatial correspondence between in-vivo MRI and histology and allows for the spatially local and quantitative assessment of pathological correlates in MRI

Quantitative MRI correlates of hippocampal and neocortical pathology in intractable temporal lobe epilepsy

Intractable or drug-resistant epilepsy occurs in over 30% of epilepsy patients, with many of these patients undergoing surgical excision of the affected brain region to achieve seizure control. Advances in MRI have the potential to improve surgical treatment of epilepsy through improved identification and delineation of lesions. However, validation is currently needed to investigate histopathological correlates of these new imaging techniques. The purpose of this work is to investigate histopathological correlates of quantitative relaxometry and DTI from hippocampal and neocortical specimens of intractable TLE patients. To achieve this goal I developed and evaluated a pipeline for histology to in-vivo MRI image registration, which finds dense spatial correspondence between both modalities. This protocol was divided in two steps whereby sparsely sectioned histology from temporal lobe specimens was first registered to the intermediate ex-vivo MRI which is then registered to the in-vivo MRI, completing a pipeline for histology to in-vivo MRI registration. When correlating relaxometry and DTI with neuronal density and morphology in the temporal lobe neocortex, I found T1 to be a predictor of neuronal density in the neocortical GM and demonstrated that employing multi-parametric MRI (combining T1 and FA together) provided a significantly better fit than each parameter alone in predicting density of neurons. This work was the first to relate in-vivo T1 and FA values to the proportion of neurons in GM. When investigating these quantitative multimodal parameters with histological features within the hippocampal subfields, I demonstrated that MD correlates with neuronal density and size, and can act as a marker for neuron integrity within the hippocampus. More importantly, this work was the first to highlight the potential of subfield relaxometry and diffusion parameters (mainly T2 and MD) as well as volumetry in predicting the extent of cell loss per subfield pre-operatively, with a precision so far unachievable. These results suggest that high-resolution quantitative MRI sequences could impact clinical practice for pre-operative evaluation and prediction of surgical outcomes of intractable epilepsy

Registering Histological and MR Images of Prostate for Image-based Cancer Detection

Rationale and Objectives Needle biopsy is currently the only way to confirm prostate cancer. To increase prostate cancer diagnostic rate, needles are expected to be deployed at suspicious cancer locations. High contrast MR imaging provides a powerful tool for detecting suspicious cancerous tissues. To do this, MR appearances of cancerous tissue should be characterized and learned from a sufficient number of prostate MR images with known cancer information. However, ground-truth cancer information is only available in histological images. Therefore, it is necessary to warp ground-truth cancerous regions in histological images to MR images by a registration procedure. The objective of this paper is to develop a registration technique for aligning histological and MR images of the same prostate. Material and Methods Five pairs of histological and T2-weighted MR images of radical prostatectomy specimens are collected. For each pair, registration is guided by two sets of correspondences that can be reliably established on prostate boundaries and internal salient blob-like structures of histological and MR images. Results Our developed registration method can accurately register histological and MR images. It yields results comparable to manual registration, in terms of landmark distance and volume overlap. It also outperforms both affine registration and boundary-guided registration methods. Conclusions We have developed a novel method for deformable registration of histological and MR images of the same prostate. Besides the collection of ground-truth cancer information in MR images, the method has other potential applications. An automatic, accurate registration of histological and MR images actually builds a bridge between in vivo anatomical information and ex vivo pathological information, which is valuable for various clinical studies

Image registration of ex-vivo MRI to sparsely sectioned histology of hippocampal and neocortical temporal lobe specimens.

Intractable or drug-resistant epilepsy occurs in up to 30% of epilepsy patients, with many of these patients undergoing surgical excision of the affected brain region to achieve seizure control. Recent magnetic resonance imaging (MRI) sequences and analysis techniques have the potential to detect abnormalities not identified with diagnostic MRI protocols. Prospective studies involving pre-operative imaging and collection of surgically-resected tissue provide a unique opportunity for verification and tuning of these image analysis techniques, since direct comparison can be made against histopathology, and can lead to better prediction of surgical outcomes and potentially less invasive procedures. To carry out MRI and histology comparison, spatial correspondence between the MR images and the histology images must be found. Towards this goal, a novel pipeline is presented here for bringing ex-vivo MRI of surgically-resected temporal lobe specimens and digital histology into spatial correspondence. The sparsely-sectioned histology images represent a challenge for 3D reconstruction which we address with a combined 3D and 2D registration algorithm that alternates between slice-based and volume-based registration with the ex-vivo MRI. We evaluated our registration method on specimens resected from patients undergoing anterior temporal lobectomy (N=7) and found our method to have a mean target registration error of 0.76±0.66 and 0.98±0.60 mm for hippocampal and neocortical specimens respectively. This work allows for the spatially-local comparison of histology with post-operative MRI and paves the way for eventual correlation with pre-operative MRI image analysis techniques