Comparative Analysis of Tissue Reconstruction Algorithms for 3D Histology

Motivation: Digital pathology enables new approaches that expand beyond storage, visualization or analysis of histological samples in digital format. One novel opportunity is 3D histology, where a three-dimensional reconstruction of the sample is formed computationally based on serial tissue sections. This allows examining tissue architecture in 3D, for example, for diagnostic purposes. Importantly, 3D histology enables joint mapping of cellular morphology with spatially resolved omics data in the true 3D context of the tissue at microscopic resolution. Several algorithms have been proposed for the reconstruction task, but a quantitative comparison of their accuracy is lacking. Results: We developed a benchmarking framework to evaluate the accuracy of several free and commercial 3D reconstruction methods using two whole slide image datasets. The results provide a solid basis for further development and application of 3D histology algorithms and indicate that methods capable of compensating for local tissue deformation are superior to simpler approaches.publishedVersionPeer reviewe

Virtual reality for 3D histology: multi-scale visualization of organs with interactive feature exploration

Virtual reality (VR) enables data visualization in an immersive and engaging manner, and it can be used for creating ways to explore scientific data. Here, we use VR for visualization of 3D histology data, creating a novel interface for digital pathology. Our contribution includes 3D modeling of a whole organ and embedded objects of interest, fusing the models with associated quantitative features and full resolution serial section patches, and implementing the virtual reality application. Our VR application is multi-scale in nature, covering two object levels representing different ranges of detail, namely organ level and sub-organ level. In addition, the application includes several data layers, including the measured histology image layer and multiple representations of quantitative features computed from the histology. In this interactive VR application, the user can set visualization properties, select different samples and features, and interact with various objects. In this work, we used whole mouse prostates (organ level) with prostate cancer tumors (sub-organ objects of interest) as example cases, and included quantitative histological features relevant for tumor biology in the VR model. Due to automated processing of the histology data, our application can be easily adopted to visualize other organs and pathologies from various origins. Our application enables a novel way for exploration of high-resolution, multidimensional data for biomedical research purposes, and can also be used in teaching and researcher training

Spa-RQ: an Image Analysis Tool to Visualise and Quantify Spatial Phenotypes Applied to Non-Small Cell Lung Cancer

To facilitate analysis of spatial tissue phenotypes, we created an open-source tool package named 'Spa-RQ' for 'Spatial tissue analysis: image Registration & Quantification'. Spa-RQ contains software for image registration (Spa-R) and quantitative analysis of DAB staining overlap (Spa-Q). It provides an easy-to-implement workflow for serial sectioning and staining as an alternative to multiplexed techniques. To demonstrate Spa-RQ's applicability, we analysed the spatial aspects of oncogenic KRAS-related signalling activities in non-small cell lung cancer (NSCLC). Using Spa-R in conjunction with ImageJ/Fiji, we first performed annotation-guided tumour-by-tumour phenotyping using multiple signalling markers. This analysis showed histopathology-selective activation of PI3K/AKT and MAPK signalling in Kras mutant murine tumours, as well as high p38MAPK stress signalling in p53 null murine NSCLC. Subsequently, Spa-RQ was applied to measure the co-activation of MAPK, AKT, and their mutual effector mTOR pathway in individual tumours. Both murine and clinical NSCLC samples could be stratified into 'MAPK/mTOR', 'AKT/mTOR', and 'Null' signature subclasses, suggesting mutually exclusive MAPK and AKT signalling activities. Spa-RQ thus provides a robust and easy to use tool that can be employed to identify spatially-distributed tissue phenotypes

Spa-RQ : an Image Analysis Tool to Visualise and Quantify Spatial Phenotypes Applied to Non-Small Cell Lung Cancer

To facilitate analysis of spatial tissue phenotypes, we created an open-source tool package named 'Spa-RQ' for 'Spatial tissue analysis: image Registration & Quantification'. Spa-RQ contains software for image registration (Spa-R) and quantitative analysis of DAB staining overlap (Spa-Q). It provides an easy-to-implement workflow for serial sectioning and staining as an alternative to multiplexed techniques. To demonstrate Spa-RQ's applicability, we analysed the spatial aspects of oncogenic KRAS-related signalling activities in non-small cell lung cancer (NSCLC). Using Spa-R in conjunction with ImageJ/Fiji, we first performed annotation-guided tumour-by-tumour phenotyping using multiple signalling markers. This analysis showed histopathology-selective activation of PI3K/AKT and MAPK signalling in Kras mutant murine tumours, as well as high p38MAPK stress signalling in p53 null murine NSCLC. Subsequently, Spa-RQ was applied to measure the co-activation of MAPK, AKT, and their mutual effector mTOR pathway in individual tumours. Both murine and clinical NSCLC samples could be stratified into 'MAPK/mTOR', 'AKT/mTOR', and 'Null' signature subclasses, suggesting mutually exclusive MAPK and AKT signalling activities. Spa-RQ thus provides a robust and easy to use tool that can be employed to identify spatially-distributed tissue phenotypes.Peer reviewe

Development of Microstructural Segmentation and 3D Reconstruction Method Using Serial Section of Tissue: 3D Educational Model of Human Hypothalamus

학위논문(석사)--서울대학교 대학원 :의과대학 의학과,2019. 8. 황영일최형진.INTRODUCTION: The 3D reconstruction technique of tissue staining images is very valuable in that it visualizes the microstructure information that Magnetic resonance imaging (MRI) and Computed tomography (CT) data cannot provide and is widely used for pathological diagnosis. Organizational 3D reconstruction needs the latest devices and software for each phase. However, in reality, it is not easy to equip all of the most brand-new equipment, and software, so the existing research has been done by only a limited number of people. For this reason, in this study, we tried to develop a 3D reconstruction method of the organization by using available laboratory equipment and highly accessible software. The human hypothalamus is relatively small compared to other brain structures, but it is the backbone of homeostasis regulation and an important structure directly linked to survival. It consists of more than 13 nuclei and microstructures, and many attempts have been made to identify them. However, most reported histology images were based on the 2D map, that cause researchers are experiencing difficulties in understanding spatial structure perception. In addition, most of the currently reported hypothalamic 3D maps are based on MRI data. This DICOM based medical image has the disadvantage that it is difficult to understand the detailed microstructure of the nucleus. In order to overcome these drawbacks, this study aims to develop a detailed 3D model of the human hypothalamus by using easily accessible devices and software. Since the 3D map of the human hypothalamus has not been reported so far, we have developed a method that allows a wider range of researchers to perform the 3D reconstruction of the tissue, which had previously been done by a limited number of people. We also tried to make the model created by the researchers easily accessible to the field. Methods: Nissl staining of human brain hypothalamus obtained by autopsy was converted to a digital image using a tissue slide scanner. The whole slide image was converted by using image processing software to adjust the resolution and extension. After that, segmentation of the hypothalamic microstructure was performed in the Adobe Photoshop software, and the missing slide images were prepared by manual interpolation. All the structure segmented images were transformed into black and white to produce a mask suitable for 3D reconstruction, and they were classified by structure. Then, the whole image was subjected to bit number correction and extension conversion suitable for 3D reconstruction using ImageJ software. Then 3D reconstruction software reconstructs the segmented structures into three dimensions and attempts 3D rendering. After transforming them into STL extensions, we tried to edit them using MeshMixer software. Through this process, 3D map was created with WebGL, and the 3D map education model of the human hypothesis was created. Results: A total of 100 staining images were obtained by Nissl staining using human brain hypothalamus. To make our results more clearly, hypothalamic 2D maps obtained in this study were compared with Allen atlas. A total of 23 segmentations were carried out including hypothalamic surrounding structures and nucleus distribution patterns. A total of 11 excluded slides were supplemented by manual interpolation. The hypothalamus 2D map was used to reconstruct the human hypothalamus as a 3D reconstructed volume model and a 3D reconstructed surface model. The 3D reconstruction surface model was obtained by using MeshMixer to complement the smoothing and the outlier point of each structure. Then, I created a hypothalamus 3D reconstruction education model using WebGL service to make possible for anyone to easily access and learn without the constraint of time and space. Discussion: In this study, I developed a method for producing 2D map and 3D reconstructed images of Nissl stained using hypothalamus tissue. This is the first 3D reconstruction model based on the hypothalamus, which is meant to help other researchers and medical personnel in education and research. Previous studies have shown that the spacing of the slices of the hypothalamus tissue was not constant, but this study succeeded in acquiring the results of the staining of the hypothalamus tissue at 100 ㎛ intervals as the basic data for 3D reconstruction. Many other types of missing images were found due to the lack of consideration of various variables that occurred during the reconstruction process. The anatomical structure and various parameters were considered and corrected for more satisfactory results. In addition, existing image-based software provides automatic segmentation function considering only the distinctive features of shaded images, so it is very inappropriate to classify subtle clustering patterns such as nucleus and structures in the human hypothalamus. It is significant that the progress process is segmented, and the separate software suitable for each process is applied, and the process of working with them is compatible with each other. Most software is free, low cost and easy to learn and use, so it provides a way to easily create an organization 3D image without expensive software or equipment. The existing hypothalamus training data were mostly 2D illustrations, but the 3D reconstructed images produced in this study are easy to grasp the positional relationship of structures more space. In particular, since the hypothalamus does not contain data showing nuclear reconstruction as a 3D reconstructed image, the educational model of this study will be of great help to many hypothalamus researchers. And the 3D WebGL education model has pedagogical value because it enables free access and access through users personal device, enabling ubiquitous learning that is not restricted by time and space. Conclusions: Through this study, I have established a method for producing 2D map and 3D reconstruction using human hypothalamus. Through the 3D reconstruction image and the education model, the positional relation of the human hypothalamus can be recognized by spatial perception. This result is pedagogically worthy because it can be used as U-learning material to help researchers self-directed learning by opening it to open source WebGL for easy use by anyone.서론: 조직 염색 영상의 3차원 재구성 기술은 MRI와 CT가 제공하지 못하는 미세구조를 시각화 하여 3차원 조직학에 활용된다는 점에서 가치가 있다. 조직 3차원 재구성에는 각 단계에 적합한 기기와 소프트웨어가 사용된다. 그러나 고가의 기기와 소프트웨어를 전부 갖추기란 쉽지 않으므로 기존의 연구는 제한적인 소수에 의해 이루어져 왔다. 사람 시상하부는 다른 뇌 구조물에 비해 상대적으로 크기는 작지만 항상성 조절의 중추이며 생존과 직결된 신체 활동을 조절하는 중요한 기관이다. 기존 시상하부에 대한 시각적 연구는 조직학 영상 기반의 2차원 지도 중심으로 이루어져 구조를 공간지각적으로 이해하는 데 많은 어려움이 있었다. 또한 현재 발표된 대부분의 사람 시상하부 3차원 지도는 MRI 데이터를 기반으로 작성되어 있어 신경핵 단위의 미세구조 정보를 충분히 전달하지 못한다. 따라서 본 연구에서는 두가지 목표를 달성하고자 하였다. 첫째로, 접근성이 좋은 장비와 소프트웨어를 활용하여 보다 넓은 범위의 연구자들이 수행할 수 있는 조직의 3차원 재구성 방법을 개발하고자 하였다. 둘째로, 위의 과정을 통해 확립한 방법을 시상하부에 적용하여 해당 분야 연구자들이 학습 및 교육에 활용할 수 있는 3차원 모델을 제작하고자 하였다. 연구 대상 및 방법: 사람 시상하부 전체와 시각로가 포함된 조직을 대상으로 조직 3차원 재구성을 시도하였다. 염색된 각 조직절편을 슬라이드 스캐너를 이용해 디지털 영상으로 변환하였고 ZEN을 사용하여 해상도 조절과 확장자 변환을 시행하였다. 전체 영상을 Adobe Photoshop을 이용하여 시각로와 미세혈관, 안쪽후각겉질의 외곽선을 기준으로 정합 하였다. 이 후 미세조직 구역화, 소실된 슬라이드 영상의 수동 보간법 적용, 전체 구역화 영상의 흑백 변환, 마스크 제작, 구조물 별 분류를 시행하였다. 또한 전체 영상을 ImageJ를 사용하여 bit수 교정 및 확장자 변환을 수행하였다. 이 후 MEDIP에서 구역화 한 구조물 영상의 3차원 재구성, STL 확장자 변환 및 내보내기를 시행하였다. 이 후 MeshMixer를 사용하여 표면의 요철과 이상점을 교정한 뒤 webGL 교육모델로 제작하였다. 이렇게 수립된 protocol을 사람 시상하부에 적용하여 내부 신경핵과 미세구조를 3차원으로 재구성하였다. 결과: Zen, Adobe Photoshop, ImageJ를 사용하여 조직 염색 영상, 시각로 2차원 지도, 색면 레이어, 패스영역 레이어, 흑백변환 영상, 흑백 반전 영상, Raw data mask를 생성하였다. 이 후 MEDIP과 Meshmixer 를 사용하여 전체조직과 시각로 부피모델, 표면모델, 교육모델을 생성하였다. 이를 사람 시상하부 미세구조의 시각화에 적용하여 조직 염색 영상, 2차원 지도, 미세구조와 신경핵 구역화 색면 레이어, 패스영역 레이어, 흑백 변환 영상, 흑백 반전 영상, 3차원 Raw data mask, 3차원 재구성 부피 모델, 표면 모델, 3차원 교육모델을 생성하였다. 고찰: 본 연구에서는 Zen, Adobe Photoshop, ImageJ, MEDIP, Meshmixer로 이어지는 조직의 3차원 재구성 제작 protocol을 개발하였다. 본 연구는 기존의 3차원 재구성 방법론과 비교했을 때 외곽선이 뚜렷하지 않은 구조물의 영상 위에 수동으로 구역화를 수행했다는 점에서 차별성이 있다. 또한 기존의 조직 3차원 재구성 방법론과 비교했을 때 각 단계에 사용되는 고가의 소프트웨어와 기기를 여러 개의 접근성이 높은 소프트웨어로 분할 및 적용했다는 점에서 기존 연구와 다르다. 본 연구의 2차원 지도는 Allen atlas가 제공하는 사람 뇌 2차원 지도와 비교했을 때 보다 촘촘한 100 ㎛의 영상을 일정한 간격으로 획득했다는 점에서 차별성이 있다. 또한 MRI 기반 3차원 재구성 모델이 제공하지 못하는 여러 미세구조와 신경핵을 시각화 했다는 점에서 가치가 있다. 기존의 2차원 신경해부학 교육자료와 비교했을 때 본 연구에서 제작한 3차원 교육모델은 구조물들의 위치관계를 공간지각적으로 보다 쉽게 파악할 수 있다는 장점이 있다. 또한 본 연구의 결과물은 기존의 3차원 신경해부학 교육자료와 달리 실험을 통해 얻은 실물 자료를 기반으로 제작되었다는 점에서 가치가 있다. 결론: 본 연구에서는 사람 시상하부 조직을 매개로 조직의 3차원 재구성 protocol을 확립하였다. 이를 통해 사람 시상하부 내부의 미세구조와 신경핵의 위치관계를 공간지각적으로 파악할 수 있는 부피모델과 표면모델을 생성할 수 있었다. 이 결과물들은 의료인 교육에 적합한 교육모델로 변환하여 공개 자료로 제공되었다.초 록 ……………………………………………………………ⅰ 목 차 …………………………………………………………… ⅳ 표 목록 ……………………………………………………… v 그림 목록…………………………………………………… vi 서 론…………………………………………………………… 9 본 론 …………………………………………………………… 15 Chapter 1. 조직의 3차원 재구성 제작 protocol 개발...... 15 Chapter 2. 사람 시상하부 조직의 3차원 재구성.............. 52 결 론…………………………………………………………… 95 참고문헌…………………………………………………………96 Abstract………………………………………………………101Maste

Artificial intelligence in histopathology image analysis for cancer precision medicine

In recent years, there have been rapid advancements in the field of computational pathology. This has been enabled through the adoption of digital pathology workflows that generate digital images of histopathological slides, the publication of large data sets of these images and improvements in computing infrastructure. Objectives in computational pathology can be subdivided into two categories, first the automation of routine workflows that would otherwise be performed by pathologists and second the addition of novel capabilities. This thesis focuses on the development, application, and evaluation of methods in this second category, specifically the prediction of gene expression from pathology images and the registration of pathology images among each other. In Study I, we developed a computationally efficient cluster-based technique to perform transcriptome-wide predictions of gene expression in prostate cancer from H&E-stained whole-slide-images (WSIs). The suggested method outperforms several baseline methods and is non-inferior to single-gene CNN predictions, while reducing the computational cost with a factor of approximately 300. We included 15,586 transcripts that encode proteins in the analysis and predicted their expression with different modelling approaches from the WSIs. In a cross-validation, 6,618 of these predictions were significantly associated with the RNA-seq expression estimates with FDR-adjusted p-values <0.001. Upon validation of these 6,618 expression predictions in a held-out test set, the association could be confirmed for 5,419 (81.9%). Furthermore, we demonstrated that it is feasible to predict the prognostic cell-cycle progression score with a Spearman correlation to the RNA-seq score of 0.527 [0.357, 0.665]. The objective of Study II is the investigation of attention layers in the context of multiple-instance-learning for regression tasks, exemplified by a simulation study and gene expression prediction. We find that for gene expression prediction, the compared methods are not distinguishable regarding their performance, which indicates that attention mechanisms may not be superior to weakly supervised learning in this context. Study III describes the results of the ACROBAT 2022 WSI registration challenge, which we organised in conjunction with the MICCAI 2022 conference. Participating teams were ranked on the median 90th percentile of distances between registered and annotated target landmarks. Median 90th percentiles for eight teams that were eligible for ranking in the test set consisting of 303 WSI pairs ranged from 60.1 µm to 15,938.0 µm. The best performing method therefore has a score slightly below the median 90th percentile of distances between first and second annotator of 67.0 µm. Study IV describes the data set that we published to facilitate the ACROBAT challenge. The data set is available publicly through the Swedish National Data Service SND and consists of 4,212 WSIs from 1,153 breast cancer patients. Study V is an example of the application of WSI registration for computational pathology. In this study, we investigate the possibility to register invasive cancer annotations from H&E to KI67 WSIs and then subsequently train cancer detection models. To this end, we compare the performance of models optimised with registered annotations to the performance of models that were optimised with annotations generated for the KI67 WSIs. The data set consists of 272 female breast cancer cases, including an internal test set of 54 cases. We find that in this test set, the performance of both models is not distinguishable regarding performance, while there are small differences in model calibration

Hyperspectral imaging for resection margin assessment during breast cancer surgery

Complete tumor removal during breast-conserving surgery remains challenging due to the lack of optimal intraoperative margin assessment techniques. This thesis investigates the potential of hyperspectral imaging to assess the resection margin during surgery. Hyperspectral imaging is a non-invasive, optical imaging technique that measures differences in the optical properties of tissue. These differences in optical properties are measured in the form of diffuse reflectance spectra and can be used to differentiate tumor from healthy tissue. By imaging and analyzing the resection margin of a specimen during surgery, direct feedback can be given to the surgeon.We started our research with imaging breast tissue slices, that were obtained after gross-sectioning lumpectomy specimen. We developed a registration method to obtain a high correlation of these optical measurements with histopathology and, thereby, created an extensive hyperspectral database that was used to research the maximum capability of hyperspectral imaging to differentiate tissue types. The highest classification results were obtained using both the visual and near-infrared wavelength range. On hyperspectral signals, representing a single tissue type, we report a sensitivity and specificity above 98%, which indicates that the optical differences in tissue composition and morphology can be used to distinguish tumor from healthy breast tissue. On hyperspectral signals, representing a mixture of tissue classes, the sensitivity and specificity decrease to 80% and 93%, respectively. This is related to the percentage of a specific tissue class in the measured volume. The next step was to image lumpectomy specimen during surgery to verify the feasibility of using hyperspectral imaging during surgery. Hyperspectral imaging was fast and could provide feedback over the entire resection surface of one side of the specimen in 3 minutes. In combination with the classification performance on the tissue slices, these findings support that hyperspectral imaging can become a powerful tool for margin assessment during breast-conserving surgery. Original promotion date: April 24, 2020 (COVID-19)<br/