Building RadiologyNET: Unsupervised annotation of a large-scale multimodal medical database

Background and objective: The usage of machine learning in medical diagnosis and treatment has witnessed significant growth in recent years through the development of computer-aided diagnosis systems that are often relying on annotated medical radiology images. However, the availability of large annotated image datasets remains a major obstacle since the process of annotation is time-consuming and costly. This paper explores how to automatically annotate a database of medical radiology images with regard to their semantic similarity. Material and methods: An automated, unsupervised approach is used to construct a large annotated dataset of medical radiology images originating from Clinical Hospital Centre Rijeka, Croatia, utilising multimodal sources, including images, DICOM metadata, and narrative diagnoses. Several appropriate feature extractors are tested for each of the data sources, and their utility is evaluated using k-means and k-medoids clustering on a representative data subset. Results: The optimal feature extractors are then integrated into a multimodal representation, which is then clustered to create an automated pipeline for labelling a precursor dataset of 1,337,926 medical images into 50 clusters of visually similar images. The quality of the clusters is assessed by examining their homogeneity and mutual information, taking into account the anatomical region and modality representation. Conclusion: The results suggest that fusing the embeddings of all three data sources together works best for the task of unsupervised clustering of large-scale medical data, resulting in the most concise clusters. Hence, this work is the first step towards building a much larger and more fine-grained annotated dataset of medical radiology images

DeepRadiologyNet: radiologist level pathology detection in CT head images

We describe a system to automatically filter clinically significant findings from computerized tomography (CT) head scans, operating at performance levels exceeding that of practicing radiologists. Our system, named DeepRadiologyNet, builds on top of deep convolutional neural networks (CNNs) trained using approximately 3.5 million CT head images gathered from over 24,000 studies taken from January 1, 2015 to August 31, 2015 and January 1, 2016 to April 30 2016 in over 80 clinical sites. For our initial system, we identified 30 phenomenological traits to be recognized in the CT scans. To test the system, we designed a clinical trial using over 4.8 million CT head images (29,925 studies), completely disjoint from the training and validation set, interpreted by 35 US Board Certified radiologists with specialized CT head experience. We measured clinically significant error rates to ascertain whether the performance of DeepRadiologyNet was comparable to or better than that of US Board Certified radiologists. DeepRadiologyNet achieved a clinically significant miss rate of 0.0367% on automatically selected high-confidence studies. Thus, DeepRadiologyNet enables significant reduction in the workload of human radiologists by automatically filtering studies and reporting on the high-confidence ones at an operating point well below the literal error rate for US Board Certified radiologists, estimated at 0.82%

Deep learning driven radiographic classification of primary bone tumors using attention augmented hybrid models

Accurate classification of primary bone tumors is necessary for timely diagnosis and effective treatment planning, particularly given the complex radiographic heterogeneity exhibited by tumor subtypes. The present study introduces two novel deep learning models, including a Convolutional Neural Network Transformer (CNNT) hybrid and a Residual Network 50 (ResNet50) model, augmented by a Convolutional Block Attention Module (CBAM) to enhance feature discrimination and contextual understanding in radiographic images. The models are trained and validated on the Bone Tumor X-ray Radiograph Dataset (BTXRD) dataset of 3,746 labeled radiographs containing nine tumor subtypes. To counter the effects of noise and class imbalance, advanced preprocessing methods like Block Matching 3D Filtering (BM3D) and data balancing using the Synthetic Minority Over sampling Technique (SMOTE) are employed. Extensive testing demonstrates that our approaches outperform current state of the art models, such as ResNet50, EfficientNet version B3 (EfficientNet-b3), You Only Look Once version 8 classification (YOLOv8s-cls), and Deep Supervision Network (DS-Net). Specifically, the ResNet50-CBAM architecture achieves an F1-score of 0.9759, an AUC-ROC score of 0.984, mean accuracy of CBAM 97.41% and a Cohen's Kappa score of 0.9718, outperforming existing benchmarks for binary tumor classification. The CNNT model also achieves competitive performance, reaching an F1-score of 0.9595 with an accuracy of 92.56%. Incorporating attention mechanisms and dataset guided preprocessing renders this framework appropriate for practical clinical settings. The findings of this research have significant implications for the healthcare sector by introducing a scalable, interpretable, and highly accurate Artificial Intelligence (AI) based diagnostic system that can support radiologists in timely diagnoses and decision making processes, ultimately contributing to better patient outcomes and alleviating the diagnostic burden in musculoskeletal oncology.</p

Lessons learned from RadiologyNET foundation models for transfer learning in medical radiology

Deep learning models require large amounts of annotated data, which are hard to obtain in the medical field, as the annotation process is laborious and depends on expert knowledge. This data scarcity hinders a model's ability to generalise effectively on unseen data, and recently, foundation models pretrained on large datasets have been proposed as a promising solution. RadiologyNET is a custom medical dataset that comprises 1,902,414 medical images covering various body parts and modalities of image acquisition. We used the RadiologyNET dataset to pretrain several popular architectures (ResNet18, ResNet34, ResNet50, VGG16, EfficientNetB3, EfficientNetB4, InceptionV3, DenseNet121, MobileNetV3Small and MobileNetV3Large). We compared the performance of ImageNet and RadiologyNET foundation models against training from randomly initialiased weights on several publicly available medical datasets: (i) Segmentation-LUng Nodule Analysis Challenge, (ii) Regression-RSNA Pediatric Bone Age Challenge, (iii) Binary classification-GRAZPEDWRI-DX and COVID-19 datasets, and (iv) Multiclass classification-Brain Tumor MRI dataset. Our results indicate that RadiologyNET-pretrained models generally perform similarly to ImageNet models, with some advantages in resource-limited settings. However, ImageNet-pretrained models showed competitive performance when fine-tuned on sufficient data. The impact of modality diversity on model performance was tested, with the results varying across tasks, highlighting the importance of aligning pretraining data with downstream applications. Based on our findings, we provide guidelines for using foundation models in medical applications and publicly release our RadiologyNET-pretrained models to support further research and development in the field. The models are available at https://github.com/AIlab-RITEH/RadiologyNET-TL-models

Medical Image Repository Clustering Based on DICOM Tags

PACS repozitoriji, koje cesto odrzavaju bolnički centri, sadrže veliki broj podataka prikupljenih kroz razne medicinske procedure kao što su CT i MR. Medicinske procedure se sastoje od slika, kao i metaoznaka povezanih uz tu sliku, najčešće u DICOM datotekama. Ovaj rad istražuje mogućnost iskorištavanja DICOM metaoznaka za automatsku anotaciju PACS baza podataka koristeći metode grupiranja. Motivacija za ovo istraživanje je evaluacija mogućnosti grupiranja semantički sličnih medicinskih slika. Podaci su prikupljeni iz dijela RadiologyNet baze podataka, koja je nastala 2017. godine. Zbog velikog broja podataka, ovo istraživanje je u obzir uzelo mali podskup te baze podataka (5%; 600000 slika). Nakon analize i pripremne obrade podataka, medicinske slike su grupiranje koristeći algoritam K-sredina s različitim brojem grupa. Prikazana je i detaljnija vizualizacija za tri različita modela (K = 25; 50; 200), a ona je uzela u obzir homogenost medicinskih slika unutar grupe kao i heterogenost između grupa. Rezultati pokazuju kako su DICOM metaoznake informativne i mogu se koristiti za grupiranje medicinskih slika.PACS repositories, owned and maintained by numerous clinical centres, contain vast amounts of recorded data from various medical procedures, namely CT and MR. Medical procedures are comprised of images as well as metadata related to that image,often using DICOM format. This thesis explores the possibility of utilising DICOM tags for automatic annotation of PACS databases, using ordinary clustering. The motivation behind this research is to evaluate the possibility of grouping semantically similar images. Data has been collected as a part of RadiologyNet database, which was built in 2017. Because of the overwhelming size of the dataset, only a small subset was used (5%; 600000 images). Following data analysis and preprocessing, K-means clustering was utilised for varying number of clusters. A detailed visualisation was done for three diferent models (K = 25; 50; 200) with regards to semantic homogeneity within clusters, and the heterogeneity between clusters. The results suggest that DICOM tags are informative and can be used for semantic clustering of medical images

Medical Image Repository Clustering Based on DICOM Tags

PACS repozitoriji, koje cesto odrzavaju bolnički centri, sadrže veliki broj podataka prikupljenih kroz razne medicinske procedure kao što su CT i MR. Medicinske procedure se sastoje od slika, kao i metaoznaka povezanih uz tu sliku, najčešće u DICOM datotekama. Ovaj rad istražuje mogućnost iskorištavanja DICOM metaoznaka za automatsku anotaciju PACS baza podataka koristeći metode grupiranja. Motivacija za ovo istraživanje je evaluacija mogućnosti grupiranja semantički sličnih medicinskih slika. Podaci su prikupljeni iz dijela RadiologyNet baze podataka, koja je nastala 2017. godine. Zbog velikog broja podataka, ovo istraživanje je u obzir uzelo mali podskup te baze podataka (5%; 600000 slika). Nakon analize i pripremne obrade podataka, medicinske slike su grupiranje koristeći algoritam K-sredina s različitim brojem grupa. Prikazana je i detaljnija vizualizacija za tri različita modela (K = 25; 50; 200), a ona je uzela u obzir homogenost medicinskih slika unutar grupe kao i heterogenost između grupa. Rezultati pokazuju kako su DICOM metaoznake informativne i mogu se koristiti za grupiranje medicinskih slika.PACS repositories, owned and maintained by numerous clinical centres, contain vast amounts of recorded data from various medical procedures, namely CT and MR. Medical procedures are comprised of images as well as metadata related to that image,often using DICOM format. This thesis explores the possibility of utilising DICOM tags for automatic annotation of PACS databases, using ordinary clustering. The motivation behind this research is to evaluate the possibility of grouping semantically similar images. Data has been collected as a part of RadiologyNet database, which was built in 2017. Because of the overwhelming size of the dataset, only a small subset was used (5%; 600000 images). Following data analysis and preprocessing, K-means clustering was utilised for varying number of clusters. A detailed visualisation was done for three diferent models (K = 25; 50; 200) with regards to semantic homogeneity within clusters, and the heterogeneity between clusters. The results suggest that DICOM tags are informative and can be used for semantic clustering of medical images