Histopathological image analysis : a review

Over the past decade, dramatic increases in computational power and improvement in image analysis algorithms have allowed the development of powerful computer-assisted analytical approaches to radiological data. With the recent advent of whole slide digital scanners, tissue histopathology slides can now be digitized and stored in digital image form. Consequently, digitized tissue histopathology has now become amenable to the application of computerized image analysis and machine learning techniques. Analogous to the role of computer-assisted diagnosis (CAD) algorithms in medical imaging to complement the opinion of a radiologist, CAD algorithms have begun to be developed for disease detection, diagnosis, and prognosis prediction to complement the opinion of the pathologist. In this paper, we review the recent state of the art CAD technology for digitized histopathology. This paper also briefly describes the development and application of novel image analysis technology for a few specific histopathology related problems being pursued in the United States and Europe

PET/MRI 및 MR-IGRT를 위한 MRI 기반 합성 CT 생성의 타당성 연구

학위논문 (박사) -- 서울대학교 대학원 : 의과대학 의과학과, 2020. 8. 이재성.Over the past decade, the application of magnetic resonance imaging (MRI) in the field of diagnosis and treatment has increased. MRI provides higher soft-tissue contrast, especially in the brain, abdominal organ, and bone marrow without the expose of ionizing radiation. Hence, simultaneous positron emission tomography/MR (PET/MR) system and MR-image guided radiation therapy (MR-IGRT) system has recently been emerged and currently available for clinical study. One major issue in PET/MR system is attenuation correction from MRI scans for PET quantification and a similar need for the assignment of electron densities to MRI scans for dose calculation can be found in MR-IGRT system. Because the MR signals are related to the proton density and relaxation properties of tissue, not to electron density. To overcome this problem, the method called synthetic CT (sCT), a pseudo CT derived from MR images, has been proposed. In this thesis, studies on generating synthetic CT and investigating the feasibility of using a MR-based synthetic CT for diagnostic and radiotherapy application were presented. Firstly, MR image-based attenuation correction (MR-AC) method using level-set segmentation for brain PET/MRI was developed. To resolve conventional inaccuracy MR-AC problem, we proposed an improved ultrashort echo time MR-AC method that was based on a multiphase level-set algorithm with main magnetic field inhomogeneity correction. We also assessed the feasibility of level-set based MR-AC method, compared with CT-AC and MR-AC provided by the manufacturer of the PET/MRI scanner. Secondly, we proposed sCT generation from the low field MR images using 2D convolution neural network model for MR-IGRT system. This sCT images were compared to the deformed CT generated using the deformable registration being used in the current system. We assessed the feasibility of using sCT for radiation treatment planning from each of the patients with pelvic, thoraic and abdominal region through geometric and dosimetric evaluation.지난 10년간 진단 및 치료분야에서 자기공명영상(Magnetic resonance imaging; MRI) 의 적용이 증가하였다. MRI는 CT와 비교해 추가적인 전리방사선의 피폭없이 뇌, 복부 기관 및 골수 등에서 더 높은 연조직 대비를 제공한다. 따라서 MRI를 적용한 양전자방출단층촬영(Positron emission tomography; PET)/MR 시스템과 MR 영상 유도 방사선 치료 시스템(MR-image guided radiation therapy; MR-IGRT)이 진단 및 치료 방사선분야에 등장하여 임상에 사용되고 있다. PET/MR 시스템의 한 가지 주요 문제는 PET 정량화를 위한 MRI 스캔으로부터의 감쇠 보정이며, MR-IGRT 시스템에서 선량 계산을 위해 MR 영상에 전자 밀도를 할당하는 것과 비슷한 필요성을 찾을 수 있다. 이는 MR 신호가 전자 밀도가 아닌 조직의 양성자 밀도 및 T1, T2 이완 특성과 관련이 있기 때문이다. 이 문제를 극복하기 위해, MR 이미지로부터 유래된 가상의 CT인 합성 CT라 불리는 방법이 제안되었다. 본 학위논문에서는 합성 CT 생성 방법 및 진단 및 방사선 치료에 적용을 위한 MR 영상 기반 합성 CT 사용의 임상적 타당성을 조사하였다. 첫째로, 뇌 PET/MR를 위한 레벨셋 분할을 이용한 MR 이미지 기반 감쇠 보정 방법을 개발하였다. MR 이미지 기반 감쇠 보정의 부정확성은 정량화 오류와 뇌 PET/MRI 연구에서 병변의 잘못된 판독으로 이어진다. 이 문제를 해결하기 위해, 자기장 불균일 보정을 포함한 다상 레벨셋 알고리즘에 기초한 개선된 초단파 에코 시간 MR-AC 방법을 제안하였다. 또한 CT-AC 및 PET/MRI 스캐너 제조업체가 제공한 MR-AC와 비교하여 레벨셋 기반 MR-AC 방법의 임상적 사용가능성을 평가하였다. 둘째로, MR-IGRT 시스템을 위한 심층 컨볼루션 신경망 모델을 사용하여 저필드 MR 이미지에서 생성된 합성 CT 방법를 제안하였다. 이 합성 CT 이미지를 변형 정합을 사용하여 생성된 변형 CT와 비교 하였다. 또한 골반, 흉부 및 복부 환자에서의 기하학적, 선량적 분석을 통해 방사선 치료계획에서의 합성 CT를 사용가능성을 평가하였다.Chapter 1. Introduction 1 1.1. Background 1 1.1.1. The Integration of MRI into Other Medical Devices 1 1.1.2. Chanllenges in the MRI Integrated System 4 1.1.3. Synthetic CT Generation 5 1.2. Purpose of Research 6 Chapter 2. MRI-based Attenuation Correction for PET/MRI 8 2.1. Background 8 2.2. Materials and Methods 10 2.2.1. Brain PET Dataset 19 2.2.2. MR-Based Attenuation Map using Level-Set Algorithm 12 2.2.3. Image Processing and Reconstruction 18 2.3. Results 20 2.4. Discussion 28 Chapter 3. MRI-based synthetic CT generation for MR-IGRT 30 3.1. Background 30 3.2. Materials and Methods 32 3.2.1. MR-dCT Paired DataSet 32 3.2.2. Synthetic CT Generation using 2D CNN 36 3.2.3. Data Analysis 38 3.3. Results 41 3.3.1. Image Comparison 41 3.3.2. Geometric Analysis 49 3.3.3. Dosimetric Analysis 49 3.4. Discussion 56 Chapter 4. Conclusions 59 Bibliography 60 Abstract in Korean (국문 초록) 64Docto

To deform or not: treatment-aware longitudinal registration for breast DCE-MRI during neoadjuvant chemotherapy via unsupervised keypoints detection

Clinicians compare breast DCE-MRI after neoadjuvant chemotherapy (NAC) with pre-treatment scans to evaluate the response to NAC. Clinical evidence supports that accurate longitudinal deformable registration without deforming treated tumor regions is key to quantifying tumor changes. We propose a conditional pyramid registration network based on unsupervised keypoint detection and selective volume-preserving to quantify changes over time. In this approach, we extract the structural and the abnormal keypoints from DCE-MRI, apply the structural keypoints for the registration algorithm to restrict large deformation, and employ volume-preserving loss based on abnormal keypoints to keep the volume of the tumor unchanged after registration. We use a clinical dataset with 1630 MRI scans from 314 patients treated with NAC. The results demonstrate that our method registers with better performance and better volume preservation of the tumors. Furthermore, a local-global-combining biomarker based on the proposed method achieves high accuracy in pathological complete response (pCR) prediction, indicating that predictive information exists outside tumor regions. The biomarkers could potentially be used to avoid unnecessary surgeries for certain patients. It may be valuable for clinicians and/or computer systems to conduct follow-up tumor segmentation and response prediction on images registered by our method. Our code is available on \url{https://github.com/fiy2W/Treatment-aware-Longitudinal-Registration}

Intrasubject multimodal groupwise registration with the conditional template entropy

Image registration is an important task in medical image analysis. Whereas most methods are designed for the registration of two images (pairwise registration), there is an increasing interest in simultaneously aligning more than two images using groupwise registration. Multimodal registration in a groupwise setting remains difficult, due to the lack of generally applicable similarity metrics. In this work, a novel similarity metric for such groupwise registration problems is proposed. The metric calculates the sum of the conditional entropy between each image in the group and a representative template image constructed iteratively using principal component analysis. The proposed metric is validated in extensive experiments on synthetic and intrasubject clinical image data. These experiments showed equivalent or improved registration accuracy compared to other state-of-the-art (dis)similarity metrics and improved transformation consistency compared to pairwise mutual information

X\mathcal{X}-Metric: An N-Dimensional Information-Theoretic Framework for Groupwise Registration and Deep Combined Computing

This paper presents a generic probabilistic framework for estimating the statistical dependency and finding the anatomical correspondences among an arbitrary number of medical images. The method builds on a novel formulation of the $N$-dimensional joint intensity distribution by representing the common anatomy as latent variables and estimating the appearance model with nonparametric estimators. Through connection to maximum likelihood and the expectation-maximization algorithm, an information\hyp{}theoretic metric called $\mathcal{X}$-metric and a co-registration algorithm named $\mathcal{X}$-CoReg are induced, allowing groupwise registration of the $N$ observed images with computational complexity of $\mathcal{O}(N)$. Moreover, the method naturally extends for a weakly-supervised scenario where anatomical labels of certain images are provided. This leads to a combined\hyp{}computing framework implemented with deep learning, which performs registration and segmentation simultaneously and collaboratively in an end-to-end fashion. Extensive experiments were conducted to demonstrate the versatility and applicability of our model, including multimodal groupwise registration, motion correction for dynamic contrast enhanced magnetic resonance images, and deep combined computing for multimodal medical images. Results show the superiority of our method in various applications in terms of both accuracy and efficiency, highlighting the advantage of the proposed representation of the imaging process

Differently stained whole slide image registration technique with landmark validation

Abstract. One of the most significant features in digital pathology is to compare and fuse successive differently stained tissue sections, also called slides, visually. Doing so, aligning different images to a common frame, ground truth, is required. Current sample scanning tools enable to create images full of informative layers of digitalized tissues, stored with a high resolution into whole slide images. However, there are a limited amount of automatic alignment tools handling large images precisely in acceptable processing time. The idea of this study is to propose a deep learning solution for histopathology image registration. The main focus is on the understanding of landmark validation and the impact of stain augmentation on differently stained histopathology images. Also, the developed registration method is compared with the state-of-the-art algorithms which utilize whole slide images in the field of digital pathology. There are previous studies about histopathology, digital pathology, whole slide imaging and image registration, color staining, data augmentation, and deep learning that are referenced in this study. The goal is to develop a learning-based registration framework specifically for high-resolution histopathology image registration. Different whole slide tissue sample images are used with a resolution of up to 40x magnification. The images are organized into sets of consecutive, differently dyed sections, and the aim is to register the images based on only the visible tissue and ignore the background. Significant structures in the tissue are marked with landmarks. The quality measurements include, for example, the relative target registration error, structural similarity index metric, visual evaluation, landmark-based evaluation, matching points, and image details. These results are comparable and can be used also in the future research and in development of new tools. Moreover, the results are expected to show how the theory and practice are combined in whole slide image registration challenges. DeepHistReg algorithm will be studied to better understand the development of stain color feature augmentation-based image registration tool of this study. Matlab and Aperio ImageScope are the tools to annotate and validate the image, and Python is used to develop the algorithm of this new registration tool. As cancer is globally a serious disease regardless of age or lifestyle, it is important to find ways to develop the systems experts can use while working with patients’ data. There is still a lot to improve in the field of digital pathology and this study is one step toward it.Eri menetelmin värjättyjen virtuaalinäytelasien rekisteröintitekniikka kiintopisteiden validointia hyödyntäen. Tiivistelmä. Yksi tärkeimmistä digitaalipatologian ominaisuuksista on verrata ja fuusioida peräkkäisiä eri menetelmin värjättyjä kudosleikkeitä toisiinsa visuaalisesti. Tällöin keskenään lähes identtiset kuvat kohdistetaan samaan yhteiseen kehykseen, niin sanottuun pohjatotuuteen. Nykyiset näytteiden skannaustyökalut mahdollistavat sellaisten kuvien luonnin, jotka ovat täynnä kerroksittaista tietoa digitalisoiduista näytteistä, tallennettuna erittäin korkean resoluution virtuaalisiin näytelaseihin. Tällä hetkellä on olemassa kuitenkin vain kourallinen automaattisia työkaluja, jotka kykenevät käsittelemään näin valtavia kuvatiedostoja tarkasti hyväksytyin aikarajoin. Tämän työn tarkoituksena on syväoppimista hyväksikäyttäen löytää ratkaisu histopatologisten kuvien rekisteröintiin. Tärkeimpänä osa-alueena on ymmärtää kiintopisteiden validoinnin periaatteet sekä eri väriaineiden augmentoinnin vaikutus. Lisäksi tässä työssä kehitettyä rekisteröintialgoritmia tullaan vertailemaan muihin kirjallisuudessa esitettyihin algoritmeihin, jotka myös hyödyntävät virtuaalinäytelaseja digitaalipatologian saralla. Kirjallisessa osiossa tullaan siteeraamaan aiempia tutkimuksia muun muassa seuraavista aihealueista: histopatologia, digitaalipatologia, virtuaalinäytelasi, kuvantaminen ja rekisteröinti, näytteen värjäys, data-augmentointi sekä syväoppiminen. Tavoitteena on kehittää oppimispohjainen rekisteröintikehys erityisesti korkearesoluutioisille digitalisoiduille histopatologisille kuville. Erilaisissa näytekuvissa tullaan käyttämään jopa 40-kertaista suurennosta. Kuvat kudoksista on järjestetty eri menetelmin värjättyihin peräkkäisiin kuvasarjoihin ja tämän työn päämääränä on rekisteröidä kuvat pohjautuen ainoastaan kudosten näkyviin osuuksiin, jättäen kuvien tausta huomioimatta. Kudosten merkittävimmät rakenteet on merkattu niin sanotuin kiintopistein. Työn laatumittauksina käytetään arvoja, kuten kohteen suhteellinen rekisteröintivirhe (rTRE), rakenteellisen samankaltaisuuindeksin mittari (SSIM), sekä visuaalista arviointia, kiintopisteisiin pohjautuvaa arviointia, yhteensopivuuskohtia, ja kuvatiedoston yksityiskohtia. Nämä arvot ovat verrattavissa myös tulevissa tutkimuksissa ja samaisia arvoja voidaan käyttää uusia työkaluja kehiteltäessä. DeepHistReg metodi toimii pohjana tässä työssä kehitettävälle näytteen värjäyksen parantamiseen pohjautuvalle rekisteröintityökalulle. Matlab ja Aperio ImageScope ovat ohjelmistoja, joita tullaan hyödyntämään tässä työssä kuvien merkitsemiseen ja validointiin. Ohjelmointikielenä käytetään Pythonia. Syöpä on maailmanlaajuisesti vakava sairaus, joka ei katso ikää eikä elämäntyyliä. Siksi on tärkeää löytää uusia keinoja kehittää työkaluja, joita asiantuntijat voivat hyödyntää jokapäiväisessä työssään potilastietojen käsittelyssä. Digitaalipatologian osa-alueella on vielä paljon innovoitavaa ja tämä työ on yksi askel eteenpäin taistelussa syöpäsairauksia vastaan