Perceptual Embedding Consistency for Seamless Reconstruction of Tilewise Style Transfer

Style transfer is a field with growing interest and use cases in deep learning. Recent work has shown Generative Adversarial Networks(GANs) can be used to create realistic images of virtually stained slide images in digital pathology with clinically validated interpretability. Digital pathology images are typically of extremely high resolution, making tilewise analysis necessary for deep learning applications. It has been shown that image generators with instance normalization can cause a tiling artifact when a large image is reconstructed from the tilewise analysis. We introduce a novel perceptual embedding consistency loss significantly reducing the tiling artifact created in the reconstructed whole slide image (WSI). We validate our results by comparing virtually stained slide images with consecutive real stained tissue slide images. We also demonstrate that our model is more robust to contrast, color and brightness perturbations by running comparative sensitivity analysis tests

A comparative evaluation of image-to-image translation methods for stain transfer in histopathology

Image-to-image translation (I2I) methods allow the generation of artificial images that share the content of the original image but have a different style. With the advances in Generative Adversarial Networks (GANs)-based methods, I2I methods enabled the generation of artificial images that are indistinguishable from natural images. Recently, I2I methods were also employed in histopathology for generating artificial images of in silico stained tissues from a different type of staining. We refer to this process as stain transfer. The number of I2I variants is constantly increasing, which makes a well justified choice of the most suitable I2I methods for stain transfer challenging. In our work, we compare twelve stain transfer approaches, three of which are based on traditional and nine on GAN-based image processing methods. The analysis relies on complementary quantitative measures for the quality of image translation, the assessment of the suitability for deep learning-based tissue grading, and the visual evaluation by pathologists. Our study highlights the strengths and weaknesses of the stain transfer approaches, thereby allowing a rational choice of the underlying I2I algorithms. Code, data, and trained models for stain transfer between H&E and Masson's Trichrome staining will be made available online.Comment: 17 pages, 3 figures, 5 tables, accepted to Medical Imaging with Deep Learning (MIDL) 2023, to be published in Proceedings of Machine Learning Researc

Computational Analysis of Tumour Microenvironment in mIHC Stained Diffuse Glioma Samples

Healthcare is a sector that has been notoriously stagnant in digital innovation, nevertheless its transformation is imminent. Digital pathology is a field that is being accentuated in light of recent technological development. With capacity to conduct high-resolution tissue imaging and managing output digitally, advanced image analysis and Machine Learning can be subsequently applied. These methods provide means to for instance automating segmentation of region-of-interests, diagnosis and knowledge discovery. Brain malignancies are particularly dire with a high fatality rate and relatively high occurrence in children. Diffuse gliomas are a subtype of brain tumours whose biological behavior range from very indolent to extremely aggressive, which is reflected in grading I - IV. The brain tumour micro-environment (TME) --- local area surrounding cancerous cells with a plethora of immune cells and other structures in interaction --- has emerged as a critical regulator of brain tumour progression. Researchers are interested in immunotherapeutic treatment of brain cancer, since modern approaches are insufficient in treatment of especially the most aggressive tumours. Additionally, the TME is rendered difficult to understand. Multiplex Immunohistochemistry (mIHC) is a novel approach in effectively mapping spatial distribution of cell types in tissue samples using multiple antibodies. In this thesis, we investigate the TME in diffuse glioma mIHC samples for three patient cases with 2-3 differing tumour grades per patient. From the 18 possibilities we selected 6 antigens (markers) of interest for further analysis. In particular, we are interested in how relative proportion of positive antigens and mean distance to nearest blood vessel vary for our selected markers in tumour progression. In order to acquire desired properties, we register each corresponding image, detect nuclei, segment cells and extract structured data from region channel intensities along with their location and distance to nearest blood vessel. Our primary finding is that M2-macrophage and T cell occurrence proportions as well as their mean distance to blood vessel grow with increasing tumour grade. The results could suggest that aforementioned cell types are of low quantity in near vicinity of blood vessels in low tumour grades, and conversely with higher quantities and more homogeneous distribution in aggressive tumours. Despite the several potential error sources and non-standardized processes in the pipeline between tissue extraction and image analysis, our results support pre-existing knowledge in that M2-macrophage proportion has a positive correlation with tumour grade.Terveydenhuollon digitaalinen kehitys on ollut hidasliikkeistä muihin sektoreihin verrattuna. Tästä huolimatta, terveydenhuollon digitaalinen muunnos on välitön ja asiaan liittyvä tutkimus jatkuvaa. Digitaalinen patologia on ala, joka viime aikaisen teknologisen kehityksen myötä on korostunut. Kudoskuvantaminen korkealla resoluutiolla ja näytteiden digitaalinen hallinta on mahdollistanut kehittyneen kuvanalysiin sekä koneoppimisen soveltamisen. Nämä metodit luovat keinot esimerkiksi biologisesti merkittävien alueiden segmentointiin, diagnoosiin ja uuden tieteellisen tiedon tuottamiseen. Aivokasvaimet ovat järkyttäviä, sillä tapauskuolleisuus ja esiintymä nuorissa ovat suhteellisen korkealla. Diffuusigliomat ovat aivokasvainten alatyyppi, jonka sisältämät kasvaimet luokitellaan niiden aggressiivisuuden perusteella eri graduksiin väliltä I - IV. Kasvaimen mikroympäristö (TME), eli syöpäsolujen paikallinen ympäristö sisältäen mm. runsaasti immuunipuolustuksen soluja vuorovaikutuksessa, on osoittautunut merkittäväksi tekijäksi kasvaimen kehityksen suhteen. Aivosyövän tutkimus painottuu immunoterapeuttisiin ratkaisuihin, sillä nykyiset hoitomuodot eivät ole tarpeeksi tehokkaita etenkään kaikista aggressiivisimpien kasvainten hoidossa. Lisäksi mikroympäristö voi olla vaikea ymmärtää. Monikanavainen immunohistokemiallinen värjäys (mIHC) on uudenlainen lähestymistapa solutyyppien spatiaalijakauman kartoittamiseen kudosnäytteissä tehokkaasti hyödyntäen useita vasta-aineita. Tässä opinnäytetyössä tutkitaan diffuusigliooma mIHC-näytteitä kolmelle potilastapaukselle. Jokaista potilasta kohti on 2-3 näytettä eri kasvainlaaduista ja yhteensä 18 mIHC-kanavaa per näyte, joista 6 otettiin tarkasteluun. Tarkalleen ottaen, solutyyppien aktivaatioiden osuudet positiivisten antigeenien perusteella ja keskimääräinen etäisyys lähimpään verisuoneen jokaista ryhmää kohti lasketaan eri kasvaimen laaduissa. Tavoitteen saavuttamiseksi näytteitä vastaavat kuvat rekisteröidään, tumat tunnistetaan, solualueet segmentoidaan ja kerätään jäsenneltyä tietoa alueiden intensiteettikanavista mukaan lukien sijainti ja sijaintia vastaava etäisyys lähimpään verisuoneen. Pääasiallinen löytö on, että M2-makrofagien ja T-solujen suhteelliset osuudet sekä keskimääräinen etäisyys lähimpään verisuoneen nousevat kasvaimen ollessa aggressiivisempi. Tulokset saattavat ehdottaa, että edellämainitut solutyypit ovat vähäisiä ja verisuonten lähellä kun kasvain on hyvänlaatuinen ja vastaavasti suurimilla osuuksilla ja enemmän homogeenisesti jakautunut kun kasvain on aggressiivisempi. Useista virhelähteistä ja kudosanalyysin liittyvistä ei-standardisoiduista prosesseista huolimatta, tuloksemme tukevat ennaltatiedettyä tietoa siitä, että M2-makrofagien osuudella on positiivinen korrelaatio kasvaimen laatuun

The effect of neural network architecture on virtual H&E staining : Systematic assessment of histological feasibility

Conventional histopathology has relied on chemical staining for over a century. The staining process makes tissue sections visible to the human eye through a tedious and labor-intensive procedure that alters the tissue irreversibly, preventing repeated use of the sample. Deep learning-based virtual staining can potentially alleviate these shortcomings. Here, we used standard brightfield microscopy on unstained tissue sections and studied the impact of increased network capacity on the resulting virtually stained H&E images. Using the generative adversarial neural network model pix2pix as a baseline, we observed that replacing simple convolutions with dense convolution units increased the structural similarity score, peak signal-to-noise ratio, and nuclei reproduction accuracy. We also demonstrated highly accurate reproduction of histology, especially with increased network capacity, and demonstrated applicability to several tissues. We show that network architecture optimization can improve the image translation accuracy of virtual H&E staining, highlighting the potential of virtual staining in streamlining histopathological analysis.publishedVersionPeer reviewe

Deep Learning in Breast Cancer Imaging: A Decade of Progress and Future Directions

Breast cancer has reached the highest incidence rate worldwide among all malignancies since 2020. Breast imaging plays a significant role in early diagnosis and intervention to improve the outcome of breast cancer patients. In the past decade, deep learning has shown remarkable progress in breast cancer imaging analysis, holding great promise in interpreting the rich information and complex context of breast imaging modalities. Considering the rapid improvement in the deep learning technology and the increasing severity of breast cancer, it is critical to summarize past progress and identify future challenges to be addressed. In this paper, we provide an extensive survey of deep learning-based breast cancer imaging research, covering studies on mammogram, ultrasound, magnetic resonance imaging, and digital pathology images over the past decade. The major deep learning methods, publicly available datasets, and applications on imaging-based screening, diagnosis, treatment response prediction, and prognosis are described in detail. Drawn from the findings of this survey, we present a comprehensive discussion of the challenges and potential avenues for future research in deep learning-based breast cancer imaging.Comment: Survey, 41 page

Virtual labeling of mitochondria in living cells using correlative imaging and physics-guided deep learning

Mitochondria play a crucial role in cellular metabolism. This paper presents a novel method to visualize mitochondria in living cells without the use of fluorescent markers. We propose a physics-guided deep learning approach for obtaining virtually labeled micrographs of mitochondria from bright-field images. We integrate a microscope’s point spread function in the learning of an adversarial neural network for improving virtual labeling. We show results (average Pearson correlation 0.86) significantly better than what was achieved by state-of-the-art (0.71) for virtual labeling of mitochondria. We also provide new insights into the virtual labeling problem and suggest additional metrics for quality assessment. The results show that our virtual labeling approach is a powerful way of segmenting and tracking individual mitochondria in bright-field images, results previously achievable only for fluorescently labeled mitochondria

Machine Learning/Deep Learning in Medical Image Processing

Many recent studies on medical image processing have involved the use of machine learning (ML) and deep learning (DL). This special issue, “Machine Learning/Deep Learning in Medical Image Processing”, has been launched to provide an opportunity for researchers in the area of medical image processing to highlight recent developments made in their fields with ML/DL. Seven excellent papers that cover a wide variety of medical/clinical aspects are selected in this special issue