Recent Advances in Machine Learning Applied to Ultrasound Imaging

Machine learning (ML) methods are pervading an increasing number of fields of application because of their capacity to effectively solve a wide variety of challenging problems. The employment of ML techniques in ultrasound imaging applications started several years ago but the scientific interest in this issue has increased exponentially in the last few years. The present work reviews the most recent (2019 onwards) implementations of machine learning techniques for two of the most popular ultrasound imaging fields, medical diagnostics and non-destructive evaluation. The former, which covers the major part of the review, was analyzed by classifying studies according to the human organ investigated and the methodology (e.g., detection, segmentation, and/or classification) adopted, while for the latter, some solutions to the detection/classification of material defects or particular patterns are reported. Finally, the main merits of machine learning that emerged from the study analysis are summarized and discussed. © 2022 by the authors. Licensee MDPI, Basel, Switzerland

OPTYMALIZACJA KLASYFIKACJI OBRAZÓW ULTRASONOGRAFICZNYCH TECHNIKĄ TRANSFER LEARNING: STRATEGIE DOSTRAJANIA I WPŁYW KLASYFIKATORA NA WSTĘPNIE WYTRENOWANE WARSTWY WEWNĘTRZNE

Transfer Learning (TL) is a popular deep learning technique used in medical image analysis, especially when data is limited. It leverages pre-trained knowledge from State-Of-The-Art (SOTA) models and applies it to specific applications through Fine-Tuning (FT). However, fine-tuning large models can be time-consuming, and determining which layers to use can be challenging. This study explores different fine-tuning strategies for five SOTA models (VGG16, VGG19, ResNet50, ResNet101, and InceptionV3) pre-trained on ImageNet. It also investigates the impact of the classifier by using a linear SVM for classification. The experiments are performed on four open-access ultrasound datasets related to breast cancer, thyroid nodules cancer, and salivary glands cancer. Results are evaluated using a five-fold stratified cross-validation technique, and metrics like accuracy, precision, and recall are computed. The findings show that fine-tuning 15% of the last layers in ResNet50 and InceptionV3 achieves good results. Using SVM for classification further improves overall performance by 6% for the two best-performing models. This research provides insights into fine-tuning strategies and the importance of the classifier in transfer learning for ultrasound image classification.Transfer Learning (TL) to popularna technika głębokiego uczenia stosowana w analizie obrazów medycznych, zwłaszcza gdy ilość danych jest ograniczona. Wykorzystuje ona wstępnie wyszkoloną wiedzę z modeli State-Of-The-Art (SOTA) i zastosowanie ich do konkretnych aplikacji poprzez dostrajanie (Fine-Tuning – FT). Jednak dostrajanie dużych modeli może być czasochłonne, a określenie, których warstw użyć, może stanowić wyzwanie. W niniejszym badaniu przeanalizowano różne strategie dostrajania dla pięciu modeli SOTA (VGG16, VGG19, ResNet50, ResNet101 i InceptionV3) wstępnie wytrenowanych na ImageNet. Zbadano również wpływ klasyfikatora przy użyciu liniowej SVM do klasyfikacji. Eksperymenty przeprowadzono na czterech ogólnodostępnych zbiorach danych ultrasonograficznych związanych z rakiem piersi, rakiem guzków tarczycy i rakiem gruczołów ślinowych. Wyniki są oceniane przy użyciu techniki pięciowarstwowej walidacji krzyżowej, a wskaźniki takie jak dokładność, precyzja i odzyskiwanie są obliczane. Wyniki pokazują, że dostrojenie 15% ostatnich warstw w ResNet50 i InceptionV3 osiąga dobre wyniki. Użycie SVM do klasyfikacji dodatkowo poprawia ogólną wydajność o 6% dla dwóch najlepszych modeli. Badania te zapewniają informacje na temat strategii dostrajania i znaczenia klasyfikatora w uczeniu transferowym dla klasyfikacji obrazów ultrasonograficznych

ENAS-B: Combining ENAS with Bayesian Optimisation for Automatic Design of Optimal CNN Architectures for Breast Lesion Classification from Ultrasound Images

Efficient Neural Architecture Search (ENAS) is a recent development in searching for optimal cell structures for Convolutional Neural Network (CNN) design. It has been successfully used in various applications including ultrasound image classification for breast lesions. However, the existing ENAS approach only optimises cell structures rather than the whole CNN architecture nor its trainable hyperparameters. This paper presents a novel framework for automatic design of CNN architectures by combining strengths of ENAS and Bayesian Optimisation in two folds. Firstly, we use ENAS to search for optimal normal and reduction cells. Secondly, with the optimal cells and a suitable hyperparameter search space, we adopt Bayesian Optimisation to find the optimal depth of the network and optimal configuration of the trainable hyperparameters. To test the validity of the proposed framework, a dataset of 1,522 breast lesion ultrasound images is used for the searching and modelling. We then evaluate the robustness of the proposed approach by testing the optimized CNN model on three external datasets consisting of 727 benign and 506 malignant lesion images. We further compare the CNN model with the default ENAS-based CNN model, and then with CNN models based on the state-of-the-art architectures. The results (error rate of no more than 20.6% on internal tests and 17.3% on average of external tests) showed that the proposed framework generates robust and light CNN models

Intelligent Diagnosis of Thyroid Ultrasound Imaging Using an Ensemble of Deep Learning Methods

Background and Objectives: At present, thyroid disorders have a great incidence in the worldwide population, so the development of alternative methods for improving the diagnosis process is necessary. Materials and Methods: For this purpose, we developed an ensemble method that fused two deep learning models, one based on convolutional neural network and the other based on transfer learning. For the first model, called 5-CNN, we developed an efficient end-to-end trained model with five convolutional layers, while for the second model, the pre-trained VGG-19 architecture was repurposed, optimized and trained. We trained and validated our models using a dataset of ultrasound images consisting of four types of thyroidal images: autoimmune, nodular, micro-nodular, and normal. Results: Excellent results were obtained by the ensemble CNN-VGG method, which outperformed the 5-CNN and VGG-19 models: 97.35% for the overall test accuracy with an overall specificity of 98.43%, sensitivity of 95.75%, positive and negative predictive value of 95.41%, and 98.05%. The micro average areas under each receiver operating characteristic curves was 0.96. The results were also validated by two physicians: an endocrinologist and a pediatrician. Conclusions: We proposed a new deep learning study for classifying ultrasound thyroidal images to assist physicians in the diagnosis process

A Survey of the Impact of Self-Supervised Pretraining for Diagnostic Tasks with Radiological Images

Self-supervised pretraining has been observed to be effective at improving feature representations for transfer learning, leveraging large amounts of unlabelled data. This review summarizes recent research into its usage in X-ray, computed tomography, magnetic resonance, and ultrasound imaging, concentrating on studies that compare self-supervised pretraining to fully supervised learning for diagnostic tasks such as classification and segmentation. The most pertinent finding is that self-supervised pretraining generally improves downstream task performance compared to full supervision, most prominently when unlabelled examples greatly outnumber labelled examples. Based on the aggregate evidence, recommendations are provided for practitioners considering using self-supervised learning. Motivated by limitations identified in current research, directions and practices for future study are suggested, such as integrating clinical knowledge with theoretically justified self-supervised learning methods, evaluating on public datasets, growing the modest body of evidence for ultrasound, and characterizing the impact of self-supervised pretraining on generalization.Comment: 32 pages, 6 figures, a literature survey submitted to BMC Medical Imagin

U-Net and its variants for medical image segmentation: theory and applications

U-net is an image segmentation technique developed primarily for medical image analysis that can precisely segment images using a scarce amount of training data. These traits provide U-net with a very high utility within the medical imaging community and have resulted in extensive adoption of U-net as the primary tool for segmentation tasks in medical imaging. The success of U-net is evident in its widespread use in all major image modalities from CT scans and MRI to X-rays and microscopy. Furthermore, while U-net is largely a segmentation tool, there have been instances of the use of U-net in other applications. As the potential of U-net is still increasing, in this review we look at the various developments that have been made in the U-net architecture and provide observations on recent trends. We examine the various innovations that have been made in deep learning and discuss how these tools facilitate U-net. Furthermore, we look at image modalities and application areas where U-net has been applied.Comment: 42 pages, in IEEE Acces

Semi-supervised Learning for Real-time Segmentation of Ultrasound Video Objects: A Review

Real-time intelligent segmentation of ultrasound video object is a demanding task in the field of medical image processing and serves as an essential and critical step in image-guided clinical procedures. However, obtaining reliable and accurate medical image annotations often necessitates expert guidance, making the acquisition of large-scale annotated datasets challenging and costly. This presents obstacles for traditional supervised learning methods. Consequently, semi-supervised learning (SSL) has emerged as a promising solution, capable of utilizing unlabeled data to enhance model performance and has been widely adopted in medical image segmentation tasks. However, striking a balance between segmentation accuracy and inference speed remains a challenge for real-time segmentation. This paper provides a comprehensive review of research progress in real-time intelligent semi-supervised ultrasound video object segmentation (SUVOS) and offers insights into future developments in this area

Enhancing Breast Cancer Prediction Using Unlabeled Data

Selles väitekirjas esitatakse sildistamata andmeid kasutav süvaõppe lähenemine rinna infiltratiivse duktaalse kartsinoomi koeregioonide automaatseks klassifitseerimiseks rinnavähi patoloogilistes digipreparaatides. Süvaõppe meetodite tööpõhimõte on sarnane inimajule, mis töötab samuti mitmetel tõlgendustasanditel. Need meetodid on osutunud tulemuslikeks ka väga keerukate probleemide nagu pildiliigituse ja esemetuvastuse lahendamisel, ületades seejuures varasemate lahendusviiside efektiivsust. Süvaõppeks on aga vaja suurt hulka sildistatud andmeid, mida võib olla keeruline saada, eriti veel meditsiinis, kuna nii haiglad kui ka patsiendid ei pruugi olla nõus sedavõrd delikaatset teavet loovutama. Lisaks sellele on masinõppesüsteemide saavutatavate aina paremate tulemuste hinnaks nende süsteemide sisemise keerukuse kasv. Selle sisemise keerukuse tõttu muutub raskemaks ka nende süsteemide töö mõistmine, mistõttu kasutajad ei kipu neid usaldama. Meditsiinilisi diagnoose ei saa järgida pimesi, kuna see võib endaga kaasa tuua patsiendi tervise kahjustamise. Mudeli mõistetavuse tagamine on seega oluline viis süsteemi usaldatavuse tõstmiseks, eriti just masinõppel põhinevate mudelite laialdasel rakendamisel sellistel kriitilise tähtsusega aladel nagu seda on meditsiin. Infiltratiivne duktaalne kartsinoom on üks levinumaid ja ka agressiivsemaid rinnavähi vorme, moodustades peaaegu 80% kõigist juhtumitest. Selle diagnoosimine on patoloogidele väga keerukas ja ajakulukas ülesanne, kuna nõuab võimalike pahaloomuliste kasvajate avastamiseks paljude healoomuliste piirkondade uurimist. Samas on infiltratiivse duktaalse kartsinoomi digipatoloogias täpne piiritlemine vähi agressiivsuse hindamise aspektist ülimalt oluline. Käesolevas uurimuses kasutatakse konvolutsioonilist närvivõrku arendamaks välja infiltratiivse duktaalse kartsinoomi diagnoosimisel rakendatav pooleldi juhitud õppe skeem. Välja pakutud raamistik suurendab esmalt väikest sildistatud andmete hulka generatiivse võistlusliku võrgu loodud sünteetiliste meditsiiniliste kujutistega. Seejärel kasutatakse juba eelnevalt treenitud võrku, et selle suurendatud andmekogumi peal läbi viia kujutuvastus, misjärel sildistamata andmed sildistatakse andmesildistusalgoritmiga. Töötluse tulemusena saadud sildistatud andmeid eelmainitud konvolutsioonilisse närvivõrku sisestades saavutatakse rahuldav tulemus: ROC kõvera alla jääv pindala ja F1 skoor on vastavalt 0.86 ja 0.77. Lisaks sellele võimaldavad välja pakutud mõistetavuse tõstmise tehnikad näha ka meditsiinilistele prognooside otsuse tegemise protsessi seletust, mis omakorda teeb süsteemi usaldamise kasutajatele lihtsamaks. Käesolev uurimus näitab, et konvolutsioonilise närvivõrgu tehtud otsuseid aitab paremini mõista see, kui kasutajatele visualiseeritakse konkreetse juhtumi puhul infiltratiivse duktaalse kartsinoomi positiivse või negatiivse otsuse langetamisel süsteemi jaoks kõige olulisemaks osutunud piirkondi.The following thesis presents a deep learning (DL) approach for automatic classification of invasive ductal carcinoma (IDC) tissue regions in whole slide images (WSI) of breast cancer (BC) using unlabeled data. DL methods are similar to the way the human brain works across different interpretation levels. These techniques have shown to outperform traditional approaches of the most complex problems such as image classification and object detection. However, DL requires a broad set of labeled data that is difficult to obtain, especially in the medical field as neither the hospitals nor the patients are willing to reveal such sensitive information. Moreover, machine learning (ML) systems are achieving better performance at the cost of becoming increasingly complex. Because of that, they become less interpretable that causes distrust from the users. Model interpretability is a way to enhance trust in a system. It is a very desirable property, especially crucial with the pervasive adoption of ML-based models in the critical domains like the medical field. With medical diagnostics, the predictions cannot be blindly followed as it may result in harm to the patient. IDC is one of the most common and aggressive subtypes of all breast cancers accounting nearly 80% of them. Assessment of the disease is a very time-consuming and challenging task for pathologists, as it involves scanning large swatches of benign regions to identify an area of malignancy. Meanwhile, accurate delineation of IDC in WSI is crucial for the estimation of grading cancer aggressiveness. In the following study, a semi-supervised learning (SSL) scheme is developed using the deep convolutional neural network (CNN) for IDC diagnosis. The proposed framework first augments a small set of labeled data with synthetic medical images, generated by the generative adversarial network (GAN) that is followed by feature extraction using already pre-trained network on the larger dataset and a data labeling algorithm that labels a much broader set of unlabeled data. After feeding the newly labeled set into the proposed CNN model, acceptable performance is achieved: the AUC and the F-measure accounting for 0.86, 0.77, respectively. Moreover, proposed interpretability techniques produce explanations for medical predictions and build trust in the presented CNN. The following study demonstrates that it is possible to enable a better understanding of the CNN decisions by visualizing areas that are the most important for a particular prediction and by finding elements that are the reasons for IDC, Non-IDC decisions made by the network