Self-supervised learning methods and applications in medical imaging analysis: A survey

The scarcity of high-quality annotated medical imaging datasets is a major problem that collides with machine learning applications in the field of medical imaging analysis and impedes its advancement. Self-supervised learning is a recent training paradigm that enables learning robust representations without the need for human annotation which can be considered an effective solution for the scarcity of annotated medical data. This article reviews the state-of-the-art research directions in self-supervised learning approaches for image data with a concentration on their applications in the field of medical imaging analysis. The article covers a set of the most recent self-supervised learning methods from the computer vision field as they are applicable to the medical imaging analysis and categorize them as predictive, generative, and contrastive approaches. Moreover, the article covers 40 of the most recent research papers in the field of self-supervised learning in medical imaging analysis aiming at shedding the light on the recent innovation in the field. Finally, the article concludes with possible future research directions in the field

A multi-scale imaging approach to understand osteoarthritis development

X-ray phase-contrast imaging is an innovative and advanced imaging method. Contrary to conventional radiology, where the image contrast is primarily determined by X-ray attenuation, phase-contrast images contain additional information generated by the phase shifts or refraction of the X-rays passing through matter. The refractive effect on tissue samples is orders of magnitude higher than the absorption effect in the X-ray energy range used in biomedical imaging. This technique makes it possible to produce excellent and enhanced image contrast, particularly when examining soft biological tissues or features with similar X-ray attenuation properties. In combination with high spatial resolution detector technology and computer tomography, X-ray phase-contrast imaging has been proved to be a powerful method to examine tissue morphology and the evolution of pathologies three-dimensionally, with great detail and without the need of contrast agents. This Thesis work has focused on developing an accurate, multi-scale X-ray-based methodology for imaging and characterizing the early stages of osteoarthritis. X-ray phase-contrast images acquired at different spatial resolutions provide unprecedented insights into cartilage and the development of its degeneration, i.e., osteoarthritis. Other types of X-ray phase-contrast imaging techniques and setups using spatial resolutions ranging from micrometer down to nanometer were applied. Lower spatial resolutions allow large sample coverage and comprehensive representations, while the nanoscale analysis provides a precise depiction of anatomical details and pathological signs. X-ray phase-contrast results are correlated to data obtained, on the same specimens, by standard laboratory methods, such as histology and transmission electron microscopy. Furthermore, X-ray phase-contrast images of cartilage were acquired using different X-ray sources and results were compared in terms of image quality. It was shown that with the use of synchrotron radiation, more detailed images and much faster data acquisitions could be achieved. A second focus in this Thesis work has been the investigation of the reaction of healthy and degenerated cartilage under different physical pressures, simulating the different levels of stress to which the tissue is subject during daily movements. A specifically designed setup was used to dynamically study cartilage response to varying pressures with X-ray phase-contrast micro-computed tomography, and a fully volumetric and quantitative methodology to accurately describe the tissue morphological variations. This study revealed changes in the behavior of the cartilage cell structure, which differ between normal and osteoarthritic cartilage tissues. The third focus of this Thesis is the realization of an automated evaluation procedure for the discrimination of healthy and cartilage images with osteoarthritis. In recent years, developments in neural networks have shown that they are excellently suited for image classification tasks. The transfer learning method was applied, in which a pre-trained neural network with cartilage images is further trained and then used for classification. This enables a fast, robust and automated grouping of images with pathological findings. A neural network constructed in this way could be used as a supporting instrument in pathology. X-ray phase-contrast imaging computed tomography can provide a powerful tool for a fully 3D, highly accurate and quantitative depiction and characterization of healthy and early stage-osteoarthritic cartilage, supporting the understanding of the development of osteoarthritis.Röntgen-Phasenkontrast-Bildgebung ist eine innovative und weiterführende Bildgebungsmethode. Im Gegensatz zu herkömlichen Absorptions-Röntgenaufnahmen, wie sie in der Radiologie verwendet werden, wird der Kontrast bei dieser Methode aus dem Effekt der Phasenverschiebung oder auch Brechung der Röngtenstrahlen gebildet. Der Brechungseffekt bei Gewebeproben ist um ein Vielfaches höher als der Absorptionseffekt des elektromagnetischen Spektrums der Röntgenstrahlen. Diese Methode ermöglicht die Darstellung von großen Kontraste im Gewebe. Unter Verwendung eines hochauflösenden Detektors und in Kombination mit der Computer-Tomographie, ist Phasenkontrast-Bildgebung eine sehr gute Methode um Knorpelgewebe und Arthrose im Knorpel zu untersuchen. Diese Arbeit beschreibt primär ein Verfahren zur Darstellung arthrotischen Knorpels im Anfangsstadium. Die mit verschiedenen Auflösungen und 3D-Phasen-Kontrast-Methoden produzierten Aufnahmen ermöglichen einen noch nie dagewesenen Einblick in den Knorpel und die Entwicklung von Arthrose im Anfangsstadium. Hierbei kam die propagationsbasierte Phasenkontrastmethode mit einer Auflösung im mikrometer Bereich und die (Nano)-Holotomographie-Methode mit einer Auflösung im Submicrometer Bereich zum Einsatz. Durch Auflösung im mikrometer Bereich kann ein großes Volumen im Knorpel gescannt werden, während die Nano-Holotomographie Methode eine sehr große Detailauflösung aufweißt. Die Phasenkontrast-Aufnahmen werden mit zwei anderen wissenschaftlichen Methoden verglichen: mikroskopische Abbildungen histologisch aufgearbeiteter Knorpelproben und Aufnahmen eines Transmissionselektroskop zeigen sehr große Übereinstimmungen zur Röntgen-Phasenkontrast-Bildgebung. Desweiteren wurden Phasenkontrast-Aufnahmen von Knorpel aus unterschiedlichen Röntgenquellen verglichen. Hierbei zeigte sich, dass mit Hilfe des Teilchenbeschleunigers (Synchrotron) detailreichere und schnellere Aufnahmen erzielt werden können. Bilder aus Flüssig-Metall-Quellen zeigen sich durchaus von guter Qualität, erfordern jedoch sehr lange Aufnahmezeiten. In dieser Arbeit wird zudem das Verhalten von Knorpelgewebe, welches ein Anfangsstadium von Arthrose aufweist, unter physikalischem Druck untersucht. Hierfür wurden 3D-Computertomographie-Aufnahmen von komprimiertem Knorpelgewebe angefertig und mit Aufnahmen ohne Komprimierung verglichen. Ein quantitativer Vergleich machte Veränderungen des Verhaltens der Knorpelzellstruktur (Chondronen) sichtbar. Es konnte gezeigt werden, dass Chondrone bei arthrotischem Knorpel ein verändertes Kompressionsverhalten haben. Der dritte Fokus dieser Arbeit liegt auf der automatisierten Auswertung von Aufnahmen gesunden und arthrotischen Knorpelgewebes. Die Entwicklungen im Bereich der Neuronale Netze zeigten in den letzten Jahren, dass diese sich hervoragend für Bildklassifizierungsaufgaben eignen. Es wurde die Methode des transferierenden Lernens angewandt, bei der ein vortrainiertes Neuronales Netz mit Knorpelbildern weitertrainiert und anschließend zur Klassifizierung eingesetzt wird. Dadurch ist eine schnelle, robuste und automatisierte Gruppierung von Bildern mit pathologischen Befunden möglich. Ein derart konstruiertes Neuronales Netz könnte als unterstützendes Instrument in der Pathologie angewandt werden. Röntgen-Phasenkontrast-CT kann ein leistungsstarkes Werkzeug für eine umfassende, hochpräzise und quantitative 3D-Darstellung und Charakterisierung von gesundem Knorpel und athrotischem Knorpel im Frühstadium bieten, um das Verständnis der Entwicklung von Osteoarthritis zu erweitern

Automatic segmentation and classification methods using optical coherence tomography angiography (Octa): A review and handbook

Optical coherence tomography angiography (OCTA) is a promising technology for the non-invasive imaging of vasculature. Many studies in literature present automated algorithms to quantify OCTA images, but there is a lack of a review on the most common methods and their comparison considering multiple clinical applications (e.g., ophthalmology and dermatology). Here, we aim to provide readers with a useful review and handbook for automatic segmentation and classification methods using OCTA images, presenting a comparison of techniques found in the literature based on the adopted segmentation or classification method and on the clinical application. Another goal of this study is to provide insight into the direction of research in automated OCTA image analysis, especially in the current era of deep learning

Deep Representation Learning with Limited Data for Biomedical Image Synthesis, Segmentation, and Detection

Biomedical imaging requires accurate expert annotation and interpretation that can aid medical staff and clinicians in automating differential diagnosis and solving underlying health conditions. With the advent of Deep learning, it has become a standard for reaching expert-level performance in non-invasive biomedical imaging tasks by training with large image datasets. However, with the need for large publicly available datasets, training a deep learning model to learn intrinsic representations becomes harder. Representation learning with limited data has introduced new learning techniques, such as Generative Adversarial Networks, Semi-supervised Learning, and Self-supervised Learning, that can be applied to various biomedical applications. For example, ophthalmologists use color funduscopy (CF) and fluorescein angiography (FA) to diagnose retinal degenerative diseases. However, fluorescein angiography requires injecting a dye, which can create adverse reactions in the patients. So, to alleviate this, a non-invasive technique needs to be developed that can translate fluorescein angiography from fundus images. Similarly, color funduscopy and optical coherence tomography (OCT) are also utilized to semantically segment the vasculature and fluid build-up in spatial and volumetric retinal imaging, which can help with the future prognosis of diseases. Although many automated techniques have been proposed for medical image segmentation, the main drawback is the model's precision in pixel-wise predictions. Another critical challenge in the biomedical imaging field is accurately segmenting and quantifying dynamic behaviors of calcium signals in cells. Calcium imaging is a widely utilized approach to studying subcellular calcium activity and cell function; however, large datasets have yielded a profound need for fast, accurate, and standardized analyses of calcium signals. For example, image sequences from calcium signals in colonic pacemaker cells ICC (Interstitial cells of Cajal) suffer from motion artifacts and high periodic and sensor noise, making it difficult to accurately segment and quantify calcium signal events. Moreover, it is time-consuming and tedious to annotate such a large volume of calcium image stacks or videos and extract their associated spatiotemporal maps. To address these problems, we propose various deep representation learning architectures that utilize limited labels and annotations to address the critical challenges in these biomedical applications. To this end, we detail our proposed semi-supervised, generative adversarial networks and transformer-based architectures for individual learning tasks such as retinal image-to-image translation, vessel and fluid segmentation from fundus and OCT images, breast micro-mass segmentation, and sub-cellular calcium events tracking from videos and spatiotemporal map quantification. We also illustrate two multi-modal multi-task learning frameworks with applications that can be extended to other domains of biomedical applications. The main idea is to incorporate each of these as individual modules to our proposed multi-modal frameworks to solve the existing challenges with 1) Fluorescein angiography synthesis, 2) Retinal vessel and fluid segmentation, 3) Breast micro-mass segmentation, and 4) Dynamic quantification of calcium imaging datasets

MacularNet: Towards Fully Automated Attention-Based Deep CNN for Macular Disease Classification

AbstractIn this work, we propose an attention-based deep convolutional neural network (CNN) model as an assistive computer-aided tool to classify common types of macular diseases: age-related macular degeneration, diabetic macular edema, diabetic retinopathy, choroidal neovascularization, macular hole, and central serous retinopathy from normal macular conditions with the help of scans from optical coherence tomography (OCT) imaging. Our proposed architecture unifies refined deep pre-trained models using transfer learning with limited training data and a deformation-aware attention mechanism encoding crucial morphological variations appearing in the deformation of retinal layers, detachments from the subsequent layers, presence of fluid-filled regions, geographic atrophy, scars, cysts, drusen, to achieve superior macular imaging classification performance. The proposed attention module facilitates the base network to automatically focus on the salient features arising due to the macular structural abnormalities while suppressing the irrelevant (or no cues) regions. The superiority of our proposed method lies in the fact that it does not require any pre-processing steps such as retinal flattening, denoising, and selection of a region of interest making it fully automatic and end-to-end trainable. Additionally, it requires a reduced number of network model parameters while achieving higher diagnostic performance. Extensive experimental results, analysis on four datasets along with the ablation studies show that the proposed architecture achieves state-of-the-art performance.</jats:p

Physics-Informed Computer Vision: A Review and Perspectives

Incorporation of physical information in machine learning frameworks are opening and transforming many application domains. Here the learning process is augmented through the induction of fundamental knowledge and governing physical laws. In this work we explore their utility for computer vision tasks in interpreting and understanding visual data. We present a systematic literature review of formulation and approaches to computer vision tasks guided by physical laws. We begin by decomposing the popular computer vision pipeline into a taxonomy of stages and investigate approaches to incorporate governing physical equations in each stage. Existing approaches in each task are analyzed with regard to what governing physical processes are modeled, formulated and how they are incorporated, i.e. modify data (observation bias), modify networks (inductive bias), and modify losses (learning bias). The taxonomy offers a unified view of the application of the physics-informed capability, highlighting where physics-informed learning has been conducted and where the gaps and opportunities are. Finally, we highlight open problems and challenges to inform future research. While still in its early days, the study of physics-informed computer vision has the promise to develop better computer vision models that can improve physical plausibility, accuracy, data efficiency and generalization in increasingly realistic applications