Simultaneous automatic scoring and co-registration of hormone receptors in tumour areas in whole slide images of breast cancer tissue slides

Aims: Automation of downstream analysis may offer many potential benefits to routine histopathology. One area of interest for automation is in the scoring of multiple immunohistochemical markers in order to predict the patient's response to targeted therapies. Automated serial slide analysis of this kind requires robust registration to identify common tissue regions across sections. We present an automated method for co-localised scoring of Estrogen Receptor and Progesterone Receptor (ER/PR) in breast cancer core biopsies using whole slide images. Methods and Results: Regions of tumour in a series of fifty consecutive breast core biopsies were identified by annotation on H&E whole slide images. Sequentially cut immunohistochemical stained sections were scored manually, before being digitally scanned and then exported into JPEG 2000 format. A two-stage registration process was performed to identify the annotated regions of interest in the immunohistochemistry sections, which were then scored using the Allred system. Overall correlation between manual and automated scoring for ER and PR was 0.944 and 0.883 respectively, with 90% of ER and 80% of PR scores within in one point or less of agreement. Conclusions: This proof of principle study indicates slide registration can be used as a basis for automation of the downstream analysis for clinically relevant biomarkers in the majority of cases. The approach is likely to be improved by implantation of safeguarding analysis steps post registration

Part-to-whole Registration of Histology and MRI using Shape Elements

Image registration between histology and magnetic resonance imaging (MRI) is a challenging task due to differences in structural content and contrast. Too thick and wide specimens cannot be processed all at once and must be cut into smaller pieces. This dramatically increases the complexity of the problem, since each piece should be individually and manually pre-aligned. To the best of our knowledge, no automatic method can reliably locate such piece of tissue within its respective whole in the MRI slice, and align it without any prior information. We propose here a novel automatic approach to the joint problem of multimodal registration between histology and MRI, when only a fraction of tissue is available from histology. The approach relies on the representation of images using their level lines so as to reach contrast invariance. Shape elements obtained via the extraction of bitangents are encoded in a projective-invariant manner, which permits the identification of common pieces of curves between two images. We evaluated the approach on human brain histology and compared resulting alignments against manually annotated ground truths. Considering the complexity of the brain folding patterns, preliminary results are promising and suggest the use of characteristic and meaningful shape elements for improved robustness and efficiency.Comment: Paper accepted at ICCV Workshop (Bio-Image Computing

Automatic reconstruction from serial sections

In many experiments in biological and medical research, serial sectioning of biological material is the only way to reveal the three dimensional (3D) structure and function. For a number of reasons other 3D imaging techniques, such as CT, MRI, and confocal microscopy, are not always adequate because they cannot provide the necessary resolution or contrast, or because the specimen is too large, or because the staining techniques require sectioning. Therefore for the foreseeable future reconstruction from serial sections will remain the only method for 3D investigations in many biomedical fields. Reconstruction is a difficult problem due to the loss of 3D alignment as the sections are cut and, more seriously, the systematic and random distortion caused by the sectioning and preparation processes.Many authors have reported how serial sections can be registered by means of fiducial markers or otherwise, but there have been only a few studies of automated correction of the sectioning distortions. In this thesis solutions to the registration problem are reviewed and discussed, and a solution to the warping problem, based on image pro¬ cessing techniques and the finite element method (FEM), is presented. The aim of this project was to develop a fully automatic method of reconstruction in order to provide a 3D atlas of mouse development as part of a gene expression database. For this purpose it is not necessary to warp the object so that it is identical to the original object, but to correct local distortions in the sections in order to produce a smooth representative mouse embryo. Furthermore the use of fiducial markers was not possible because the reconstructions were from already sectioned material.In this thesis we demonstrate a new method for warping serial sections. The sections are warped by applying forces to each section, where each section is modelled as a thin elastic plate. The deformation forces are determined from correspondences between sections which are calculated by combining match strengths and positional information. The equilibrium state which represents the reconstructed 3D image is calculated using the finite element method. Results of the application of these methods to paraffin wax and resin embedded sections of the mouse embryo are presented

Registration of histology and magnetic resonance imaging of the brain

Combining histology and non-invasive imaging has been attracting the attention of the medical imaging community for a long time, due to its potential to correlate macroscopic information with the underlying microscopic properties of tissues. Histology is an invasive procedure that disrupts the spatial arrangement of the tissue components but enables visualisation and characterisation at a cellular level. In contrast, macroscopic imaging allows non-invasive acquisition of volumetric information but does not provide any microscopic details. Through the establishment of spatial correspondences obtained via image registration, it is possible to compare micro- and macroscopic information and to recover the original histological arrangement in three dimensions. In this thesis, I present: (i) a survey of the literature relative to methods for histology reconstruction with and without the help of 3D medical imaging; (ii) a graph-theoretic method for histology volume reconstruction from sets of 2D sections, without external information; (iii) a method for multimodal 2D linear registration between histology and MRI based on partial matching of shape-informative boundaries

Automated Vascular Smooth Muscle Segmentation, Reconstruction, Classification and Simulation on Whole-Slide Histology

Histology of the microvasculature depicts detailed characteristics relevant to tissue perfusion. One important histologic feature is the smooth muscle component of the microvessel wall, which is responsible for controlling vessel caliber. Abnormalities can cause disease and organ failure, as seen in hypertensive retinopathy, diabetic ischemia, Alzheimer’s disease and improper cardiovascular development. However, assessments of smooth muscle cell content are conventionally performed on selected fields of view on 2D sections, which may lead to measurement bias. We have developed a software platform for automated (1) 3D vascular reconstruction, (2) detection and segmentation of muscularized microvessels, (3) classification of vascular subtypes, and (4) simulation of function through blood flow modeling. Vessels were stained for α-actin using 3,3\u27-Diaminobenzidine, assessing both normal (n=9 mice) and regenerated vasculature (n=5 at day 14, n=4 at day 28). 2D locally adaptive segmentation involved vessel detection, skeletonization, and fragment connection. 3D reconstruction was performed using our novel nucleus landmark-based registration. Arterioles and venules were categorized using supervised machine learning based on texture and morphometry. Simulation of blood flow for the normal and regenerated vasculature was performed at baseline and during demand based on the structural measures obtained from the above tools. Vessel medial area and vessel wall thickness were found to be greater in the normal vasculature as compared to the regenerated vasculature (p\u3c0.001) and a higher density of arterioles was found in the regenerated tissue (p\u3c0.05). Validation showed: a Dice coefficient of 0.88 (compared to manual) for the segmentations, a 3D reconstruction target registration error of 4 μm, and area under the receiver operator curve of 0.89 for vessel classification. We found 89% and 67% decreases in the blood flow through the network for the regenerated vasculature during increased oxygen demand as compared to the normal vasculature, respectively for 14 and 28 days post-ischemia. We developed a software platform for automated vasculature histology analysis involving 3D reconstruction, segmentation, and arteriole vs. venule classification. This advanced the knowledge of conventional histology sampling compared to whole slide analysis, the morphological and density differences in the regenerated vasculature, and the effect of the differences on blood flow and function

Submicron-resolution Photoacoustic Microscopy of Endogenous Light-absorbing Biomolecules

Photoacoustic imaging in biomedicine has the unique advantage of probing endogenous light absorbers at various length scales with a 100% relative sensitivity. Among the several modalities of photoacoustic imaging, optical-resolution photoacoustic microscopy (OR-PAM) can achieve high spatial resolution, on the order of optical wavelength, at \u3c1 mm depth in biological tissue (the optical ballistic regime). OR-PAM has been applied successfully to structural and functional imaging of blood vasculature and red blood cells in vivo. Any molecules which absorb sufficient light at certain wavelengths can potentially be imaged by PAM. Compared with pure optical imaging, which typically targets fluorescent markers, label-free PAM avoids the major concerns that the fluorescent labeling probes may disturb the function of biomolecules and may have an insufficient density. This dissertation aims to advance label-free OR-PAM to the subcellular scale. The first part of this dissertation describes the technological advancement of PAM yielding high spatial resolution in 3D. The lateral resolution was improved by using optical objectives with high numerical apertures for optical focusing. The axial resolution was improved by using broadband ultrasonic transducers for ultrasound detection. We achieved 220 nm lateral resolution in transmission mode, 0.43 µm lateral resolution in reflection mode, 7.6 µm axial resolution in normal tissue, and 5.8 µm axial resolution with silicone oil immersion/injection. The achieved lateral resolution and axial resolution were the finest reported at the time. With high-resolution in 3D, PAM was demonstrated to resolve cellular and subcellular structures in vivo, such as red blood cells and melanosomes in melanoma cells. Compared with previous PAM systems, our high-resolution PAM could resolve capillaries in mouse ears more clearly. As an example application, we demonstrated intracellular temperature imaging, assisted by fluorescence signal detection, with sub-degree temperature resolution and sub-micron lateral resolution. The second part of this dissertation describes the exploration of endogenous light-absorbing biomolecules for PAM. We demonstrated cytochromes and myoglobin as new absorption contrasts for PAM and identified the corresponding optimal wavelengths for imaging. Fixed fibroblasts on slides and mouse ear sections were imaged by PAM at 422 nm and 250 nm wavelengths to reveal cytoplasms and nuclei, respectively, as confirmed by standard hematoxylin and eosin (H&E) histology. By imaging a blood-perfused mouse heart at 532 nm down to 150 µm in depth, we derived the myocardial sheet thickness and the cleavage height from an undehydrated heart for the first time. The findings promote PAM at new wavelengths and open up new possibilities for characterizing biological tissue. Of particular interest, dual-wavelength PAM around 250 nm and 420 nm wavelengths is analogous to H&E histology. The last part of this dissertation describes the development of sectioning photoacoustic microscopy (SPAM), based on the advancement in spatial resolution and new contrasts for PAM, with applications in brain histology. Label-free SPAM, assisted by a microtome, acquires serial distortion-free images of a specimen on the surface. By exciting cell nuclei at 266 nm wavelength with high resolution, SPAM could pinpoint cell nuclei sensitively and specifically in the mouse brain section, as confirmed by H&E histology. SPAM was demonstrated to generate high-resolution 3D images, highlighting cell nuclei, of formalin-fixed paraffin-embedded mouse brains without tissue staining or clearing. SPAM can potentially serve as a high-throughput and minimal-artifact substitute for histology, probe many other biomolecules and cells, and become a universal tool for animal or human whole-organ microscopy, with diverse applications in life sciences

Registration of pre-operative lung cancer PET/CT scans with post-operative histopathology images

Non-invasive imaging modalities used in the diagnosis of lung cancer, such as Positron Emission Tomography (PET) or Computed Tomography (CT), currently provide insuffcient information about the cellular make-up of the lesion microenvironment, unless they are compared against the gold standard of histopathology.The aim of this retrospective study was to build a robust imaging framework for registering in vivo and post-operative scans from lung cancer patients, in order to have a global, pathology-validated multimodality map of the tumour and its surroundings.;Initial experiments were performed on tissue-mimicking phantoms, to test different shape reconstruction methods. The choice of interpolator and slice thickness were found to affect the algorithm's output, in terms of overall volume and local feature recovery. In the second phase of the study, nine lung cancer patients referred for radical lobectomy were recruited. Resected specimens were inflated with agar, sliced at 5 mm intervals, and each cross-section was photographed. The tumour area was delineated on the block-face pathology images and on the preoperative PET/CT scans.;Airway segments were also added to the reconstructed models, to act as anatomical fiducials. Binary shapes were pre-registered by aligning their minimal bounding box axes, and subsequently transformed using rigid registration. In addition, histopathology slides were matched to the block-face photographs using moving least squares algorithm.;A two-step validation process was used to evaluate the performance of the proposed method against manual registration carried out by experienced consultants. In two out of three cases, experts rated the results generated by the algorithm as the best output, suggesting that the developed framework outperforms the current standard practice.Non-invasive imaging modalities used in the diagnosis of lung cancer, such as Positron Emission Tomography (PET) or Computed Tomography (CT), currently provide insuffcient information about the cellular make-up of the lesion microenvironment, unless they are compared against the gold standard of histopathology.The aim of this retrospective study was to build a robust imaging framework for registering in vivo and post-operative scans from lung cancer patients, in order to have a global, pathology-validated multimodality map of the tumour and its surroundings.;Initial experiments were performed on tissue-mimicking phantoms, to test different shape reconstruction methods. The choice of interpolator and slice thickness were found to affect the algorithm's output, in terms of overall volume and local feature recovery. In the second phase of the study, nine lung cancer patients referred for radical lobectomy were recruited. Resected specimens were inflated with agar, sliced at 5 mm intervals, and each cross-section was photographed. The tumour area was delineated on the block-face pathology images and on the preoperative PET/CT scans.;Airway segments were also added to the reconstructed models, to act as anatomical fiducials. Binary shapes were pre-registered by aligning their minimal bounding box axes, and subsequently transformed using rigid registration. In addition, histopathology slides were matched to the block-face photographs using moving least squares algorithm.;A two-step validation process was used to evaluate the performance of the proposed method against manual registration carried out by experienced consultants. In two out of three cases, experts rated the results generated by the algorithm as the best output, suggesting that the developed framework outperforms the current standard practice

Concept and Design of a Hand-held Mobile Robot System for Craniotomy

This work demonstrates a highly intuitive robot for Surgical Craniotomy Procedures. Utilising a wheeled hand-held robot, to navigate the Craniotomy Drill over a patient\u27s skull, the system does not remove the surgeons from the procedure, but supports them during this critical phase of the operation

An Adaptive Algorithm to Identify Ambiguous Prostate Capsule Boundary Lines for Three-Dimensional Reconstruction and Quantitation

Currently there are few parameters that are used to compare the efficiency of different methods of cancerous prostate surgical removal. An accurate assessment of the percentage and depth of extra-capsular soft tissue removed with the prostate by the various surgical techniques can help surgeons determine the appropriateness of surgical approaches. Additionally, an objective assessment can allow a particular surgeon to compare individual performance against a standard. In order to facilitate 3D reconstruction and objective analysis and thus provide more accurate quantitation results when analyzing specimens, it is essential to automatically identify the capsule line that separates the prostate gland tissue from its extra-capsular tissue. However the prostate capsule is sometimes unrecognizable due to the naturally occurring intrusion of muscle and connective tissue into the prostate gland. At these regions where the capsule disappears, its contour can be arbitrarily reconstructed by drawing a continuing contour line based on the natural shape of the prostate gland. Presented here is a mathematical model that can be used in deciding the missing part of the capsule. This model approximates the missing parts of the capsule where it disappears to a standard shape by using a Generalized Hough Transform (GHT) approach to detect the prostate capsule. We also present an algorithm based on a least squares curve fitting technique that uses a prostate shape equation to merge previously detected capsule parts with the curve equation to produce an approximated curve that represents the prostate capsule. We have tested our algorithms using three shapes on 13 prostate slices that are cut at different locations from the apex and the results are promisin