338 research outputs found
GAN-based Virtual Re-Staining: A Promising Solution for Whole Slide Image Analysis
Histopathological cancer diagnosis is based on visual examination of stained
tissue slides. Hematoxylin and eosin (H\&E) is a standard stain routinely
employed worldwide. It is easy to acquire and cost effective, but cells and
tissue components show low-contrast with varying tones of dark blue and pink,
which makes difficult visual assessments, digital image analysis, and
quantifications. These limitations can be overcome by IHC staining of target
proteins of the tissue slide. IHC provides a selective, high-contrast imaging
of cells and tissue components, but their use is largely limited by a
significantly more complex laboratory processing and high cost. We proposed a
conditional CycleGAN (cCGAN) network to transform the H\&E stained images into
IHC stained images, facilitating virtual IHC staining on the same slide. This
data-driven method requires only a limited amount of labelled data but will
generate pixel level segmentation results. The proposed cCGAN model improves
the original network \cite{zhu_unpaired_2017} by adding category conditions and
introducing two structural loss functions, which realize a multi-subdomain
translation and improve the translation accuracy as well. % need to give
reasons here. Experiments demonstrate that the proposed model outperforms the
original method in unpaired image translation with multi-subdomains. We also
explore the potential of unpaired images to image translation method applied on
other histology images related tasks with different staining techniques
Differently stained whole slide image registration technique with landmark validation
Abstract. One of the most significant features in digital pathology is to compare and fuse successive differently stained tissue sections, also called slides, visually. Doing so, aligning different images to a common frame, ground truth, is required. Current sample scanning tools enable to create images full of informative layers of digitalized tissues, stored with a high resolution into whole slide images. However, there are a limited amount of automatic alignment tools handling large images precisely in acceptable processing time. The idea of this study is to propose a deep learning solution for histopathology image registration. The main focus is on the understanding of landmark validation and the impact of stain augmentation on differently stained histopathology images. Also, the developed registration method is compared with the state-of-the-art algorithms which utilize whole slide images in the field of digital pathology.
There are previous studies about histopathology, digital pathology, whole slide imaging and image registration, color staining, data augmentation, and deep learning that are referenced in this study. The goal is to develop a learning-based registration framework specifically for high-resolution histopathology image registration. Different whole slide tissue sample images are used with a resolution of up to 40x magnification. The images are organized into sets of consecutive, differently dyed sections, and the aim is to register the images based on only the visible tissue and ignore the background. Significant structures in the tissue are marked with landmarks.
The quality measurements include, for example, the relative target registration error, structural similarity index metric, visual evaluation, landmark-based evaluation, matching points, and image details. These results are comparable and can be used also in the future research and in development of new tools. Moreover, the results are expected to show how the theory and practice are combined in whole slide image registration challenges. DeepHistReg algorithm will be studied to better understand the development of stain color feature augmentation-based image registration tool of this study. Matlab and Aperio ImageScope are the tools to annotate and validate the image, and Python is used to develop the algorithm of this new registration tool.
As cancer is globally a serious disease regardless of age or lifestyle, it is important to find ways to develop the systems experts can use while working with patients’ data. There is still a lot to improve in the field of digital pathology and this study is one step toward it.Eri menetelmin värjättyjen virtuaalinäytelasien rekisteröintitekniikka kiintopisteiden validointia hyödyntäen. Tiivistelmä. Yksi tärkeimmistä digitaalipatologian ominaisuuksista on verrata ja fuusioida peräkkäisiä eri menetelmin värjättyjä kudosleikkeitä toisiinsa visuaalisesti. Tällöin keskenään lähes identtiset kuvat kohdistetaan samaan yhteiseen kehykseen, niin sanottuun pohjatotuuteen. Nykyiset näytteiden skannaustyökalut mahdollistavat sellaisten kuvien luonnin, jotka ovat täynnä kerroksittaista tietoa digitalisoiduista näytteistä, tallennettuna erittäin korkean resoluution virtuaalisiin näytelaseihin. Tällä hetkellä on olemassa kuitenkin vain kourallinen automaattisia työkaluja, jotka kykenevät käsittelemään näin valtavia kuvatiedostoja tarkasti hyväksytyin aikarajoin. Tämän työn tarkoituksena on syväoppimista hyväksikäyttäen löytää ratkaisu histopatologisten kuvien rekisteröintiin. Tärkeimpänä osa-alueena on ymmärtää kiintopisteiden validoinnin periaatteet sekä eri väriaineiden augmentoinnin vaikutus. Lisäksi tässä työssä kehitettyä rekisteröintialgoritmia tullaan vertailemaan muihin kirjallisuudessa esitettyihin algoritmeihin, jotka myös hyödyntävät virtuaalinäytelaseja digitaalipatologian saralla.
Kirjallisessa osiossa tullaan siteeraamaan aiempia tutkimuksia muun muassa seuraavista aihealueista: histopatologia, digitaalipatologia, virtuaalinäytelasi, kuvantaminen ja rekisteröinti, näytteen värjäys, data-augmentointi sekä syväoppiminen. Tavoitteena on kehittää oppimispohjainen rekisteröintikehys erityisesti korkearesoluutioisille digitalisoiduille histopatologisille kuville. Erilaisissa näytekuvissa tullaan käyttämään jopa 40-kertaista suurennosta. Kuvat kudoksista on järjestetty eri menetelmin värjättyihin peräkkäisiin kuvasarjoihin ja tämän työn päämääränä on rekisteröidä kuvat pohjautuen ainoastaan kudosten näkyviin osuuksiin, jättäen kuvien tausta huomioimatta. Kudosten merkittävimmät rakenteet on merkattu niin sanotuin kiintopistein.
Työn laatumittauksina käytetään arvoja, kuten kohteen suhteellinen rekisteröintivirhe (rTRE), rakenteellisen samankaltaisuuindeksin mittari (SSIM), sekä visuaalista arviointia, kiintopisteisiin pohjautuvaa arviointia, yhteensopivuuskohtia, ja kuvatiedoston yksityiskohtia. Nämä arvot ovat verrattavissa myös tulevissa tutkimuksissa ja samaisia arvoja voidaan käyttää uusia työkaluja kehiteltäessä. DeepHistReg metodi toimii pohjana tässä työssä kehitettävälle näytteen värjäyksen parantamiseen pohjautuvalle rekisteröintityökalulle. Matlab ja Aperio ImageScope ovat ohjelmistoja, joita tullaan hyödyntämään tässä työssä kuvien merkitsemiseen ja validointiin. Ohjelmointikielenä käytetään Pythonia.
Syöpä on maailmanlaajuisesti vakava sairaus, joka ei katso ikää eikä elämäntyyliä. Siksi on tärkeää löytää uusia keinoja kehittää työkaluja, joita asiantuntijat voivat hyödyntää jokapäiväisessä työssään potilastietojen käsittelyssä. Digitaalipatologian osa-alueella on vielä paljon innovoitavaa ja tämä työ on yksi askel eteenpäin taistelussa syöpäsairauksia vastaan
Whole slide image registration for the study of tumor heterogeneity
Consecutive thin sections of tissue samples make it possible to study local
variation in e.g. protein expression and tumor heterogeneity by staining for a
new protein in each section. In order to compare and correlate patterns of
different proteins, the images have to be registered with high accuracy. The
problem we want to solve is registration of gigapixel whole slide images (WSI).
This presents 3 challenges: (i) Images are very large; (ii) Thin sections
result in artifacts that make global affine registration prone to very large
local errors; (iii) Local affine registration is required to preserve correct
tissue morphology (local size, shape and texture). In our approach we compare
WSI registration based on automatic and manual feature selection on either the
full image or natural sub-regions (as opposed to square tiles). Working with
natural sub-regions, in an interactive tool makes it possible to exclude
regions containing scientifically irrelevant information. We also present a new
way to visualize local registration quality by a Registration Confidence Map
(RCM). With this method, intra-tumor heterogeneity and charateristics of the
tumor microenvironment can be observed and quantified.Comment: MICCAI2018 - Computational Pathology and Ophthalmic Medical Image
Analysis - COMPA
Tissue Phenomics for prognostic biomarker discovery in low- and intermediate-risk prostate cancer
Tissue Phenomics is the discipline of mining tissue images to identify patterns that are related to clinical outcome providing potential prognostic and predictive value. This involves the discovery process from assay development, image analysis, and data mining to the final interpretation and validation of the findings. Importantly, this process is not linear but allows backward steps and optimization loops over multiple sub-processes. We provide a detailed description of the Tissue Phenomics methodology while exemplifying each step on the application of prostate cancer recurrence prediction. In particular, we automatically identified tissue-based biomarkers having significant prognostic value for low-and intermediate-risk prostate cancer patients (Gleason scores 6-7b) after radical prostatectomy. We found that promising phenes were related to CD8(+) and CD68(+) cells in the microenvironment of cancerous glands in combination with the local micro-vascularization. Recurrence prediction based on the selected phenes yielded accuracies up to 83% thereby clearly outperforming prediction based on the Gleason score. Moreover, we compared different machine learning algorithms to combine the most relevant phenes resulting in increased accuracies of 88% for tumor progression prediction. These findings will be of potential use for future prognostic tests for prostate cancer patients and provide a proof-of-principle of the Tissue Phenomics approach
Computational Histological Staining and Destaining of Prostate Core Biopsy RGB Images with Generative Adversarial Neural Networks
Histopathology tissue samples are widely available in two states:
paraffin-embedded unstained and non-paraffin-embedded stained whole slide RGB
images (WSRI). Hematoxylin and eosin stain (H&E) is one of the principal stains
in histology but suffers from several shortcomings related to tissue
preparation, staining protocols, slowness and human error. We report two novel
approaches for training machine learning models for the computational H&E
staining and destaining of prostate core biopsy RGB images. The staining model
uses a conditional generative adversarial network that learns hierarchical
non-linear mappings between whole slide RGB image (WSRI) pairs of prostate core
biopsy before and after H&E staining. The trained staining model can then
generate computationally H&E-stained prostate core WSRIs using previously
unseen non-stained biopsy images as input. The destaining model, by learning
mappings between an H&E stained WSRI and a non-stained WSRI of the same biopsy,
can computationally destain previously unseen H&E-stained images. Structural
and anatomical details of prostate tissue and colors, shapes, geometries,
locations of nuclei, stroma, vessels, glands and other cellular components were
generated by both models with structural similarity indices of 0.68 (staining)
and 0.84 (destaining). The proposed staining and destaining models can engender
computational H&E staining and destaining of WSRI biopsies without additional
equipment and devices.Comment: Accepted for publication at 2018 IEEE International Conference on
Machine Learning and Applications (ICMLA
Automated Vascular Smooth Muscle Segmentation, Reconstruction, Classification and Simulation on Whole-Slide Histology
Histology of the microvasculature depicts detailed characteristics relevant to tissue perfusion. One important histologic feature is the smooth muscle component of the microvessel wall, which is responsible for controlling vessel caliber. Abnormalities can cause disease and organ failure, as seen in hypertensive retinopathy, diabetic ischemia, Alzheimer’s disease and improper cardiovascular development. However, assessments of smooth muscle cell content are conventionally performed on selected fields of view on 2D sections, which may lead to measurement bias. We have developed a software platform for automated (1) 3D vascular reconstruction, (2) detection and segmentation of muscularized microvessels, (3) classification of vascular subtypes, and (4) simulation of function through blood flow modeling. Vessels were stained for α-actin using 3,3\u27-Diaminobenzidine, assessing both normal (n=9 mice) and regenerated vasculature (n=5 at day 14, n=4 at day 28). 2D locally adaptive segmentation involved vessel detection, skeletonization, and fragment connection. 3D reconstruction was performed using our novel nucleus landmark-based registration. Arterioles and venules were categorized using supervised machine learning based on texture and morphometry. Simulation of blood flow for the normal and regenerated vasculature was performed at baseline and during demand based on the structural measures obtained from the above tools. Vessel medial area and vessel wall thickness were found to be greater in the normal vasculature as compared to the regenerated vasculature (p\u3c0.001) and a higher density of arterioles was found in the regenerated tissue (p\u3c0.05). Validation showed: a Dice coefficient of 0.88 (compared to manual) for the segmentations, a 3D reconstruction target registration error of 4 μm, and area under the receiver operator curve of 0.89 for vessel classification. We found 89% and 67% decreases in the blood flow through the network for the regenerated vasculature during increased oxygen demand as compared to the normal vasculature, respectively for 14 and 28 days post-ischemia. We developed a software platform for automated vasculature histology analysis involving 3D reconstruction, segmentation, and arteriole vs. venule classification. This advanced the knowledge of conventional histology sampling compared to whole slide analysis, the morphological and density differences in the regenerated vasculature, and the effect of the differences on blood flow and function
A Colour Wheel to Rule them All: Analysing Colour & Geometry in Medical Microscopy
Personalized medicine is a rapidly growing field in healthcare that aims to customize
medical treatments and preventive measures based on each patient’s unique characteristics,
such as their genes, environment, and lifestyle factors. This approach
acknowledges that people with the same medical condition may respond differently
to therapies and seeks to optimize patient outcomes while minimizing the risk
of adverse effects.
To achieve these goals, personalized medicine relies on advanced technologies,
such as genomics, proteomics, metabolomics, and medical imaging. Digital
histopathology, a crucial aspect of medical imaging, provides clinicians with valuable
insights into tissue structure and function at the cellular and molecular levels. By
analyzing small tissue samples obtained through minimally invasive techniques, such
as biopsy or aspirate, doctors can gather extensive data to evaluate potential diagnoses
and clinical decisions. However, digital analysis of histology images presents
unique challenges, including the loss of 3D information and stain variability, which
is further complicated by sample variability. Limited access to data exacerbates
these challenges, making it difficult to develop accurate computational models for
research and clinical use in digital histology.
Deep learning (DL) algorithms have shown significant potential for improving the
accuracy of Computer-Aided Diagnosis (CAD) and personalized treatment models,
particularly in medical microscopy. However, factors such as limited generability,
lack of interpretability, and bias sometimes hinder their clinical impact. Furthermore,
the inherent variability of histology images complicates the development of robust DL
methods. Thus, this thesis focuses on developing new tools to address these issues.
Our essential objective is to create transparent, accessible, and efficient methods
based on classical principles from various disciplines, including histology, medical
imaging, mathematics, and art, to tackle microscopy image registration and colour
analysis successfully. These methods can contribute significantly to the advancement
of personalized medicine, particularly in studying the tumour microenvironment
for diagnosis and therapy research.
First, we introduce a novel automatic method for colour analysis and non-rigid
histology registration, enabling the study of heterogeneity morphology in tumour
biopsies. This method achieves accurate tissue cut registration, drastically reducing
landmark distance and excellent border overlap. Second, we introduce ABANICCO, a novel colour analysis method that combines
geometric analysis, colour theory, fuzzy colour spaces, and multi-label systems
for automatically classifying pixels into a set of conventional colour categories.
ABANICCO outperforms benchmark methods in accuracy and simplicity. It is
computationally straightforward, making it useful in scenarios involving changing
objects, limited data, unclear boundaries, or when users lack prior knowledge of
the image or colour theory. Moreover, results can be modified to match each
particular task.
Third, we apply the acquired knowledge to create a novel pipeline of rigid
histology registration and ABANICCO colour analysis for the in-depth study of
triple-negative breast cancer biopsies. The resulting heterogeneity map and tumour
score provide valuable insights into the composition and behaviour of the tumour,
informing clinical decision-making and guiding treatment strategies.
Finally, we consolidate the developed ideas into an efficient pipeline for tissue
reconstruction and multi-modality data integration on Tuberculosis infection data.
This enables accurate element distribution analysis to understand better interactions
between bacteria, host cells, and the immune system during the course of infection.
The methods proposed in this thesis represent a transparent approach to computational
pathology, addressing the needs of medical microscopy registration and
colour analysis while bridging the gap between clinical practice and computational
research. Moreover, our contributions can help develop and train better, more
robust DL methods.En una época en la que la medicina personalizada está revolucionando la asistencia
sanitaria, cada vez es más importante adaptar los tratamientos y las medidas
preventivas a la composición genética, el entorno y el estilo de vida de cada
paciente. Mediante el empleo de tecnologías avanzadas, como la genómica, la
proteómica, la metabolómica y la imagen médica, la medicina personalizada se
esfuerza por racionalizar el tratamiento para mejorar los resultados y reducir
los efectos secundarios.
La microscopía médica, un aspecto crucial de la medicina personalizada, permite
a los médicos recopilar y analizar grandes cantidades de datos a partir de pequeñas
muestras de tejido. Esto es especialmente relevante en oncología, donde las terapias
contra el cáncer se pueden optimizar en función de la apariencia tisular específica de
cada tumor. La patología computacional, un subcampo de la visión por ordenador,
trata de crear algoritmos para el análisis digital de biopsias. Sin embargo, antes de
que un ordenador pueda analizar imágenes de microscopía médica, hay que seguir
varios pasos para conseguir las imágenes de las muestras.
La primera etapa consiste en recoger y preparar una muestra de tejido del
paciente. Para que esta pueda observarse fácilmente al microscopio, se corta en
secciones ultrafinas. Sin embargo, este delicado procedimiento no está exento de
dificultades. Los frágiles tejidos pueden distorsionarse, desgarrarse o agujerearse,
poniendo en peligro la integridad general de la muestra.
Una vez que el tejido está debidamente preparado, suele tratarse con tintes de
colores característicos. Estos tintes acentúan diferentes tipos de células y tejidos
con colores específicos, lo que facilita a los profesionales médicos la identificación
de características particulares. Sin embargo, esta mejora en visualización tiene
un alto coste. En ocasiones, los tintes pueden dificultar el análisis informático
de las imágenes al mezclarse de forma inadecuada, traspasarse al fondo o alterar
el contraste entre los distintos elementos.
El último paso del proceso consiste en digitalizar la muestra. Se toman imágenes
de alta resolución del tejido con distintos aumentos, lo que permite su análisis por
ordenador. Esta etapa también tiene sus obstáculos. Factores como una calibración
incorrecta de la cámara o unas condiciones de iluminación inadecuadas pueden
distorsionar o hacer borrosas las imágenes. Además, las imágenes de porta completo
obtenidas so de tamaño considerable, complicando aún más el análisis. En general, si bien la preparación, la tinción y la digitalización de las muestras
de microscopía médica son fundamentales para el análisis digital, cada uno de estos
pasos puede introducir retos adicionales que deben abordarse para garantizar un
análisis preciso. Además, convertir un volumen de tejido completo en unas pocas
secciones teñidas reduce drásticamente la información 3D disponible e introduce
una gran incertidumbre.
Las soluciones de aprendizaje profundo (deep learning, DL) son muy prometedoras
en el ámbito de la medicina personalizada, pero su impacto clínico a veces se
ve obstaculizado por factores como la limitada generalizabilidad, el sobreajuste, la
opacidad y la falta de interpretabilidad, además de las preocupaciones éticas y en
algunos casos, los incentivos privados. Por otro lado, la variabilidad de las imágenes
histológicas complica el desarrollo de métodos robustos de DL. Para superar estos
retos, esta tesis presenta una serie de métodos altamente robustos e interpretables
basados en principios clásicos de histología, imagen médica, matemáticas y arte,
para alinear secciones de microscopía y analizar sus colores.
Nuestra primera contribución es ABANICCO, un innovador método de análisis
de color que ofrece una segmentación de colores objectiva y no supervisada y permite
su posterior refinamiento mediante herramientas fáciles de usar. Se ha demostrado
que la precisión y la eficacia de ABANICCO son superiores a las de los métodos
existentes de clasificación y segmentación del color, e incluso destaca en la detección
y segmentación de objetos completos. ABANICCO puede aplicarse a imágenes
de microscopía para detectar áreas teñidas para la cuantificación de biopsias, un
aspecto crucial de la investigación de cáncer.
La segunda contribución es un método automático y no supervisado de segmentación
de tejidos que identifica y elimina el fondo y los artefactos de las
imágenes de microscopía, mejorando así el rendimiento de técnicas más sofisticadas
de análisis de imagen. Este método es robusto frente a diversas imágenes, tinciones
y protocolos de adquisición, y no requiere entrenamiento.
La tercera contribución consiste en el desarrollo de métodos novedosos para
registrar imágenes histopatológicas de forma eficaz, logrando el equilibrio adecuado
entre un registro preciso y la preservación de la morfología local, en función de
la aplicación prevista.
Como cuarta contribución, los tres métodos mencionados se combinan para
crear procedimientos eficientes para la integración completa de datos volumétricos,
creando visualizaciones altamente interpretables de toda la información presente en
secciones consecutivas de biopsia de tejidos. Esta integración de datos puede tener
una gran repercusión en el diagnóstico y el tratamiento de diversas enfermedades,
en particular el cáncer de mama, al permitir la detección precoz, la realización
de pruebas clínicas precisas, la selección eficaz de tratamientos y la mejora en la
comunicación el compromiso con los pacientes. Por último, aplicamos nuestros hallazgos a la integración multimodal de datos y
la reconstrucción de tejidos para el análisis preciso de la distribución de elementos
químicos en tuberculosis, lo que arroja luz sobre las complejas interacciones entre
las bacterias, las células huésped y el sistema inmunitario durante la infección
tuberculosa. Este método también aborda problemas como el daño por adquisición,
típico de muchas modalidades de imagen.
En resumen, esta tesis muestra la aplicación de métodos clásicos de visión por
ordenador en el registro de microscopía médica y el análisis de color para abordar
los retos únicos de este campo, haciendo hincapié en la visualización eficaz y fácil de
datos complejos. Aspiramos a seguir perfeccionando nuestro trabajo con una amplia
validación técnica y un mejor análisis de los datos. Los métodos presentados en esta
tesis se caracterizan por su claridad, accesibilidad, visualización eficaz de los datos,
objetividad y transparencia. Estas características los hacen perfectos para tender
puentes robustos entre los investigadores de inteligencia artificial y los clínicos e
impulsar así la patología computacional en la práctica y la investigación médicas.Programa de Doctorado en Ciencia y Tecnología Biomédica por la Universidad Carlos III de MadridPresidenta: María Jesús Ledesma Carbayo.- Secretario: Gonzalo Ricardo Ríos Muñoz.- Vocal: Estíbaliz Gómez de Marisca
Translating AI to digital pathology workflow: Dealing with scarce data and high variation by minimising complexities in data and models
The recent conversion to digital pathology using Whole Slide Images (WSIs) from conventional pathology opened the doors for Artificial Intelligence (AI) in pathology workflow. The recent interests in machine learning and deep learning have gained a high interest in medical image processing. However, WSIs differ from generic medical images. WSIs are complex images which can reveal various information to support different diagnosis varying from cancer to unknown underlying conditions which were not discovered in other medical investigations. These investigations require expert knowledge spending a long time for investigations, applying different stains to the WSIs, and comparing the WSIs. Differences in WSI differentiate general machine learning methods that are applied for medical image processing. Co-analysing multistained WSIs, high variation of the WSIs from different sites, and lack of labelled data are the main key interest areas that directly influence in developing machine learning models that support pathologists in their investigations. However, most of the state-ofthe- art machine learning approaches cannot be applied in the general clinical workflow without using high compute power, expert knowledge, and time. Therefore, this thesis explores avenues to translate the highly computational and time intensive model to a clinical workflow. Co-analysing multi-stained WSIs require registering differently stained WSI together. In order to get a high precision in the registration exploring nonrigid and rigid transformation is required. The non-rigid transformation requires complex deep learning approaches. Using super-convergence on a small Convolutional Neural Network model it is possible to achieve high precision compared to larger auto-encoders and other state-of-the-art models. High variation of the WSIs from different sites heavily effect machine learning models in their predictions. The thesis presents an approach of using a pre-trained model by using only a small number of samples from the new site. Therefore, re-training larger deep learning models are not required which saves expert time for re-labelling and computational power. Finally, lack of labelled data is one of the main issues in training any supervised machine learning or deep learning model. Using a Generative Adversarial Networks (GAN) is an approach which can be easily implemented to avoid this issue. However, GANs are time and computationally expensive. These are not applicable in a general clinical workflow. Therefore, this thesis presents an approach using a simpler GANthat can generate accurate sample labelled data. The synthetic data are used to train classifier and the thesis demonstrates that the predictive model can generate higher accuracy in the test environment. This thesis demonstrates that machine learning and deep learning models can be applied to a clinical workflow, without exploiting expert time and high computing power
- …