Recent Advances in Artificial Intelligence-Assisted Ultrasound Scanning

Funded by the Spanish Ministry of Economic Affairs and Digital Transformation (Project MIA.2021.M02.0005 TARTAGLIA, from the Recovery, Resilience, and Transformation Plan financed by the European Union through Next Generation EU funds). TARTAGLIA takes place under the R&D Missions in Artificial Intelligence program, which is part of the Spain Digital 2025 Agenda and the Spanish National Artificial Intelligence Strategy.Ultrasound (US) is a flexible imaging modality used globally as a first-line medical exam procedure in many different clinical cases. It benefits from the continued evolution of ultrasonic technologies and a well-established US-based digital health system. Nevertheless, its diagnostic performance still presents challenges due to the inherent characteristics of US imaging, such as manual operation and significant operator dependence. Artificial intelligence (AI) has proven to recognize complicated scan patterns and provide quantitative assessments for imaging data. Therefore, AI technology has the potential to help physicians get more accurate and repeatable outcomes in the US. In this article, we review the recent advances in AI-assisted US scanning. We have identified the main areas where AI is being used to facilitate US scanning, such as standard plane recognition and organ identification, the extraction of standard clinical planes from 3D US volumes, and the scanning guidance of US acquisitions performed by humans or robots. In general, the lack of standardization and reference datasets in this field makes it difficult to perform comparative studies among the different proposed methods. More open-access repositories of large US datasets with detailed information about the acquisition are needed to facilitate the development of this very active research field, which is expected to have a very positive impact on US imaging.Depto. de Estructura de la Materia, Física Térmica y ElectrónicaFac. de Ciencias FísicasTRUEMinistry of Economic Affairs and Digital Transformation from the Recovery, Resilience, and Transformation PlanNext Generation EU fundspu

Analyse der Körperzusammensetzung: Messung der Skelettmuskulatur mit Computertomographie und Implikationen für die Patientenversorgung

Objective: This thesis aims to evaluate the relationship between the skeletal muscle index derived from computed tomography (CT) images and patient outcomes, as well as its implications for patient care. This goal was pursued in five individual studies: Studies A and B evaluated the relationship between the lumbar skeletal muscle index (L3SMI) and patient outcomes in the intensive care unit (ICU) and oncology setting, respectively. Studies C and D evaluated the effect of CT acquisition parameters on body composition measures. Study E proposed a novel technique to automate the segmentation of skeletal muscle using a fully automated deep learning system. Material and methods: In total, 1328 axial CT images were included in the five studies. Patients in studies A and B were part of the clinical trials NCT01967056 and NCT01401907 at Massachusetts General Hospital (MGH), respectively. Body composition indices were computed using semi-automated segmentation. Multivariable regression models with a priori defined covariates were used to analyze clinical outcomes. To evaluate whether CT acquisition parameters influence segmentation, the Bland-Altman approach was used. In study E, a fully convolutional neural network was implemented for deep learning-based automatic segmentation. Results: Study A found lower L3SMI to be a predictor of increased mortality within 30 days of extubation (p = 0.033), increased rate of pneumonia within 30 days of extubation (p = 0.002), increased adverse discharge disposition (p = 0.044), longer hospital stays post-extubation (p = 0.048), and higher total hospital costs (p = 0.043). In study B, low L3SMI was associated with worse quality of life (p = 0.048) and increased depression symptoms (p = 0.005). Threshold-based segmentation of skeletal muscle in study C and adipose tissue compartments in study D were significantly affected by CT acquisition parameters. The proposed deep learning system in study E provided automatic segmentation of skeletal muscle cross-sectional area and achieved a high congruence to segmentations performed by domain experts (average Dice coefficient of 0.93). Conclusion: L3SMI is a useful tool for the assessment of muscle mass in clinical practice. In critically ill patients, L3SMI can provide clinically useful information about patient outcomes at the time of extubation. Patients with advanced cancer who suffered from low muscle mass reported worse quality of life and increased depression symptoms. This highlights the clinical relevance of addressing muscle loss early on as part of a multimodal treatment plan. Importantly, indices utilized in body composition analysis are significantly affected by CT acquisition parameters. These effects should be considered when body composition analysis is used in clinical practice or research studies. Finally, our fully automated deep learning system enabled instantaneous segmentation of skeletal muscle.Zielsetzung: Das Ziel der vorliegenden Dissertation war es, den Einfluss des auf CT-Bildern berechneten Skelettmuskelindexes auf klinische Ergebnisse von Patienten und die daraus resultierenden Implikationen für die Patientenversorgung zu evaluieren. Dieses Ziel wurde in fünf Einzelstudien verfolgt: In den Studien A und B wurde der Einfluss des lumbalen Skelettmuskelindex (L3SMI) auf klinische Endpunkte von Patienten auf der Intensivstation sowie in der Onkologie untersucht. Die Studien C und D evaluierten die Auswirkungen von CT-Akquisitionsparametern auf Indizes der Körperzusammensetzung. Studie E stellte eine neuartige Technik der automatisierten Segmentierung von Skelettmuskulatur vor, die durch maschinelles Lernen ermöglicht wurde. Material und Methoden: Insgesamt wurden 1328 axiale CT-Bilder in die fünf Studien eingeschlossen. Die Patienten der Studien A und B waren Teilnehmer der klinischen Studien NCT01967056 und NCT01401907 am Massachusetts General Hospital. Die Indizes der Körperzusammensetzung wurden mithilfe halbautomatischer Segmentierung berechnet. Die klinischen Endpunkte wurden in multivariablen Regressionsmodellen mit a priori definierten Kovariaten analysiert. Um zu evaluieren, ob CT-Akquisitionsparameter die Segmentierung beeinflussen, wurde der Bland-Altman-Ansatz verwendet. In Studie E wurden ein künstliches neuronales Netzwerk sowie maschinelles Lernen für die automatische Segmentierung eingesetzt. Ergebnisse: In Studie A war ein niedriger L3SMI ein Prädiktor für eine höhere Mortalität (p = 0.033) und Pneumonierate (p = 0.002) innerhalb von 30 Tagen nach der Extubation sowie für mehr ungünstige Entlassungen (p = 0.044) und höhere Behandlungskosten für den gesamten Krankenhausaufenthalt (p = 0.043). Ein niedriger L3SMI war in Studie B mit einer schlechteren Lebensqualität (p = 0.048) und stärkeren depressiven Symptomen (p = 0.005) assoziiert. Die schwellenwertbasierte Segmentierung der Skelettmuskulatur in Studie C und der Fettgewebekompartimente in Studie D wurde durch CT-Akquisitionsparameter signifikant beeinflusst. Das in Studie E vorgestellte vollautomatische Segmentierungssystem erreichte eine hohe Übereinstimmung mit den durch Experten erstellten Segmentationen (durchschnittlicher Dice-Koeffizient von 0.93). Fazit: Der L3SMI ist ein Werkzeug zur Beurteilung von Muskelmasse. Bei Intensivpatienten kann L3SMI zum Zeitpunkt der Extubation nützliche klinische Informationen liefern. Patienten mit fortgeschrittener Krebserkrankung, die zudem eine geringere Muskelmasse hatten, berichteten über eine schlechtere Lebensqualität und stärkere depressive Symptome. Dies unterstreicht die Notwendigkeit, die Muskulatur frühzeitig als Teil eines multimodalen Behandlungskonzeptes zu adressieren. Die Indizes der Körperzusammensetzung werden signifikant von CT-Akquisitionsparametern beeinflusst. Darüber hinaus ermöglichte unser vollautomatisiertes System dank maschinellen Lernens die verzögerungsfreie Segmentierung von Skelettmuskulatur

A Visualization Method of Knowledge Graphs for the Computation and Comprehension of Ultrasound Reports

Knowledge graph visualization in ultrasound reports is essential for enhancing medical decision making and the efficiency and accuracy of computer-aided analysis tools. This study aims to propose an intelligent method for analyzing ultrasound reports through knowledge graph visualization. Firstly, we provide a novel method for extracting key term networks from the narrative text in ultrasound reports with high accuracy, enabling the identification and annotation of clinical concepts within the report. Secondly, a knowledge representation framework based on ultrasound reports is proposed, which enables the structured and intuitive visualization of ultrasound report knowledge. Finally, we propose a knowledge graph completion model to address the lack of entities in physicians’ writing habits and improve the accuracy of visualizing ultrasound knowledge. In comparison to traditional methods, our proposed approach outperforms the extraction of knowledge from complex ultrasound reports, achieving a significantly higher extraction index (η) of 2.69, surpassing the general pattern-matching method (2.12). In comparison to other state-of-the-art methods, our approach achieves the highest P (0.85), R (0.89), and F1 (0.87) across three testing datasets. The proposed method can effectively utilize the knowledge embedded in ultrasound reports to obtain relevant clinical information and improve the accuracy of using ultrasound knowledge

Development of probes for molecular imaging : evaluation in models of inflammation and atherosclerosis

The imaging field is rapidly evolving and in the last two decades there have been tremendous developments in the field of multimodal imaging. Multimodal molecular imaging approaches that utilize ultrasound/magnetic resonance imaging (US/MRI), single-photon emission computed tomography/computed tomography (SPECT/CT), or positron emission tomography/MRI (PET/MRI) may provide additional detailed information at the cellular and molecular level to help identify patients with vulnerable plaques that are at risk of rupture. The search for specific biomarkers in combination with specific and optimized molecular probes may help to prevent adverse events such as myocardial infarctions or strokes. Current clinical contrast agents do not provide information on the inflammatory components of atherosclerotic plaques; thus, more specific molecular probes are needed. This thesis focuses on probe development for different molecular imaging techniques using multimodal and targeting approaches. Several types of molecular probe were evaluated: bimodal and multimodal microbubbles, as well as chemically modified human serum albumin (HSA)-based probes (aconitylated (Aco) and maleylated (Mal)) for targeting markers of inflammation; adhesion molecules on endothelial cells or macrophages, and scavenger receptor A1 (SR-A1) on macrophages. Evaluation of these molecular probes was facilitated by their physical properties enabling assessment with fluorescence microscopy, flow cytometry, and nuclear imaging properties for in vivo molecular imaging with SPECT/CT and PET/MRI. We found that functionalizing molecular probes with targeting moieties greatly improved the targeting specificity and avidity to the target compared to non-targeted molecular probes. Furthermore, these molecular probes were successfully radiolabeled with a detectable in vivo signal by 99mTc-anti-ICAM-1- MBs imaging of inflammation with SPETC/CT, and atherosclerosis by 89Zr-Mal-HSA with PET/MRI. Ex vivo evaluation of HSA-based probes showed significant accumulation in atherosclerotic lesions of Apoe-/- mice, as quantified by gamma counter and phosphor imaging autoradiography, compared to wild type (WT) mice. In conclusion, adhesion molecule targeting and scavenger receptor targeting with functionally modified probes in this thesis showed potential for the imaging of inflammation and atherosclerosis. Of the evaluated probes, modified HSA-based probes seem to have the greatest potential for clinical application in molecular imaging of atherosclerosis

Medical image retrieval for augmenting diagnostic radiology

Even though the use of medical imaging to diagnose patients is ubiquitous in clinical settings, their interpretations are still challenging for radiologists. Many factors make this interpretation task difficult, one of which is that medical images sometimes present subtle clues yet are crucial for diagnosis. Even worse, on the other hand, similar clues could indicate multiple diseases, making it challenging to figure out the definitive diagnoses. To help radiologists quickly and accurately interpret medical images, there is a need for a tool that can augment their diagnostic procedures and increase efficiency in their daily workflow. A general-purpose medical image retrieval system can be such a tool as it allows them to search and retrieve similar cases that are already diagnosed to make comparative analyses that would complement their diagnostic decisions. In this thesis, we contribute to developing such a system by proposing approaches to be integrated as modules of a single system, enabling it to handle various information needs of radiologists and thus augment their diagnostic processes during the interpretation of medical images. We have mainly studied the following retrieval approaches to handle radiologists’different information needs; i) Retrieval Based on Contents, ii) Retrieval Based on Contents, Patients’ Demographics, and Disease Predictions, and iii) Retrieval Based on Contents and Radiologists’ Text Descriptions. For the first study, we aimed to find an effective feature representation method to distinguish medical images considering their semantics and modalities. To do that, we have experimented different representation techniques based on handcrafted methods (mainly texture features) and deep learning (deep features). Based on the experimental results, we propose an effective feature representation approach and deep learning architectures for learning and extracting medical image contents. For the second study, we present a multi-faceted method that complements image contents with patients’ demographics and deep learning-based disease predictions, making it able to identify similar cases accurately considering the clinical context the radiologists seek. For the last study, we propose a guided search method that integrates an image with a radiologist’s text description to guide the retrieval process. This method guarantees that the retrieved images are suitable for the comparative analysis to confirm or rule out initial diagnoses (the differential diagnosis procedure). Furthermore, our method is based on a deep metric learning technique and is better than traditional content-based approaches that rely on only image features and, thus, sometimes retrieve insignificant random images

Theranostics in radiology : development of targeted contrast media with treatment capability

Imaging is essential in the diagnostics and medicine of today. The development of new contrast agents is important for obtaining specific information from images and to distinguish disease. Microbubbles (MB) have previously been introduced as a contrast agent for ultrasound. By incorporating super paramagnetic iron oxide nanoparticles (SPION) to the polymer matrix of the MB or between its shell layers we obtain a contrast media for Magnetic Resonance Imaging (MRI); while functionalizing the MB by ligands for labeling with 99mTc enables imaging using Single-Photon Emission Tomography (SPECT). The use of hybrid SPECT- and Computed Tomography (CT) or MRI systems enables fusion of the images from the different modalities to obtain SPECT/CT or SPECT/MR images. In the research underlying this thesis we investigated the preclinical characteristics, biodistribution and kinetics of several types of MB in Sprague Dawley rats by injecting single- and multiple layer SPION MB as well as ligand functionalized- and SPION MB labeled with 99mTc. The results obtained from imaging was correlated and compared to the histopathology of MB findings in organs. Moreover, mice were injected with Alexa-680 Vivo Tag labeled MB for imaging using a pre-clinical In Vivo Imaging System (IVIS)/μCT. Sprague Dawley rats (300 ± 50 g) were injected with single layer SPION-, multiple layer SPION-, 99mTc-labeled ligand functionalized diethylenetriamine penta-acetic acid (DTPA)-, thiolated poly(methacrylic acid) (PMAA)-, chitosan-, 1,4,7- triazacyclononane-1,4,7-triacetic acid (NOTA)-, NOTA-SPION- or DTPA-SPION MB intravenously (i.v.) through the tail vein. The rats injected with SPION MB were scanned using MRI, while the rats injected with 99mTc-labeled DTPA-, PMAA-, chitosan- or NOTA MB were scanned using SPECT/CT. The rats injected with NOTA- SPION- or DTPA-SPION MB were co-registrated using SPECT/CT and MRI. The organs from rats injected with the nuclear medicine marker were removed post mortem and measured for radioactivity. The rats injected with SPION MB were sacrificed and their organs were removed post mortem for histopathology examination using Perls’ Prussian blue staining to show iron content and immunohistochemistry (IHC) to visualize macrophage uptake of MB. Mice (30 ± 5 g) were injected with multiple layer fluorescence Alexa-680 MB and imaged using IVIS. Their organs were removed post mortem and examined using pathology and the fluorescence of MB was visualized under the microscope. The uptake of MB was mainly seen in the lungs and liver 1-2 h post-injection, while the main distribution of MB at 24 h post-injection was seen in the liver. In conclusion the MB matrix can be functionalized by ligands, labeled by SPION, 99mTc and fluorescence Alexa-680 Vivo Tag to enable its visualization in vivo using multimodal imaging SPECT/CT, SPECT/MRI or IVIS/μCT. Furthermore we have shown that MB can be loaded with cytostatic- or inflammatory drugs for theranostics. Future studies regarding MB should address toxicity and efficiency in drug loading and delivery

ChatRadio-Valuer: A Chat Large Language Model for Generalizable Radiology Report Generation Based on Multi-institution and Multi-system Data

Radiology report generation, as a key step in medical image analysis, is critical to the quantitative analysis of clinically informed decision-making levels. However, complex and diverse radiology reports with cross-source heterogeneity pose a huge generalizability challenge to the current methods under massive data volume, mainly because the style and normativity of radiology reports are obviously distinctive among institutions, body regions inspected and radiologists. Recently, the advent of large language models (LLM) offers great potential for recognizing signs of health conditions. To resolve the above problem, we collaborate with the Second Xiangya Hospital in China and propose ChatRadio-Valuer based on the LLM, a tailored model for automatic radiology report generation that learns generalizable representations and provides a basis pattern for model adaptation in sophisticated analysts' cases. Specifically, ChatRadio-Valuer is trained based on the radiology reports from a single institution by means of supervised fine-tuning, and then adapted to disease diagnosis tasks for human multi-system evaluation (i.e., chest, abdomen, muscle-skeleton, head, and maxillofacial $\&$ neck) from six different institutions in clinical-level events. The clinical dataset utilized in this study encompasses a remarkable total of \textbf{332,673} observations. From the comprehensive results on engineering indicators, clinical efficacy and deployment cost metrics, it can be shown that ChatRadio-Valuer consistently outperforms state-of-the-art models, especially ChatGPT (GPT-3.5-Turbo) and GPT-4 et al., in terms of the diseases diagnosis from radiology reports. ChatRadio-Valuer provides an effective avenue to boost model generalization performance and alleviate the annotation workload of experts to enable the promotion of clinical AI applications in radiology reports

Parametric Imaging of Contrast-Enhanced Ultrasound (CEUS) for the Evaluation of Acute Gastrointestinal Graft-Versus-Host Disease

In recent years contrast-enhanced ultrasound (CEUS) has been an emerging diagnostic modality for the detection of acute gastrointestinal (GI) graft-versus-host disease (GvHD) in patients after allogeneic stem cell transplantation. However, broad clinical usage has been partially limited by its high dependence on the expertise of an experienced examiner. Thus, the aim of this study was to facilitate detection of acute GI GvHD by implementing false color-coded parametric imaging of CEUS. As such, two inexperienced examiners with basic knowledge in abdominal and vascular ultrasound analyzed parametric images obtained from patients with clinical suspicion for acute GvHD in a blinded fashion. As diagnostic gold standard, histopathological GvHD severity score on intestinal biopsies obtained from lower GI tract endoscopy was performed. The evaluation of parametric images by the two inexperienced ultrasound examiners in patients with histological confirmation of acute GI GvHD was successful in 17 out of 19 patients (89%) as opposed to analysis of combined B-mode ultrasound, strain elastography, and CEUS by an experienced examiner, which was successful in 18 out of 19 of the patients (95%). Therefore, CEUS with parametric imaging of the intestine was technically feasible and has the potential to become a valuable diagnostic tool for rapid and widely accessible detection of acute GvHD in clinical practice