Curriculum Labeling: Revisiting Pseudo-Labeling for Semi-Supervised Learning

In this paper we revisit the idea of pseudo-labeling in the context of semi-supervised learning where a learning algorithm has access to a small set of labeled samples and a large set of unlabeled samples. Pseudo-labeling works by applying pseudo-labels to samples in the unlabeled set by using a model trained on the combination of the labeled samples and any previously pseudo-labeled samples, and iteratively repeating this process in a self-training cycle. Current methods seem to have abandoned this approach in favor of consistency regularization methods that train models under a combination of different styles of self-supervised losses on the unlabeled samples and standard supervised losses on the labeled samples. We empirically demonstrate that pseudo-labeling can in fact be competitive with the state-of-the-art, while being more resilient to out-of-distribution samples in the unlabeled set. We identify two key factors that allow pseudo-labeling to achieve such remarkable results (1) applying curriculum learning principles and (2) avoiding concept drift by restarting model parameters before each self-training cycle. We obtain 94.91% accuracy on CIFAR-10 using only 4,000 labeled samples, and 68.87% top-1 accuracy on Imagenet-ILSVRC using only 10% of the labeled samples. The code is available at https://github.com/uvavision/Curriculum-LabelingComment: In the 35th AAAI Conference on Artificial Intelligence. AAAI 202

Deep active learning for suggestive segmentation of biomedical image stacks via optimisation of Dice scores and traced boundary length

Manual segmentation of stacks of 2D biomedical images (e.g., histology) is a time-consuming task which can be sped up with semi-automated techniques. In this article, we present a suggestive deep active learning framework that seeks to minimise the annotation effort required to achieve a certain level of accuracy when labelling such a stack. The framework suggests, at every iteration, a specific region of interest (ROI) in one of the images for manual delineation. Using a deep segmentation neural network and a mixed cross-entropy loss function, we propose a principled strategy to estimate class probabilities for the whole stack, conditioned on heterogeneous partial segmentations of the 2D images, as well as on weak supervision in the form of image indices that bound each ROI. Using the estimated probabilities, we propose a novel active learning criterion based on predictions for the estimated segmentation performance and delineation effort, measured with average Dice scores and total delineated boundary length, respectively, rather than common surrogates such as entropy. The query strategy suggests the ROI that is expected to maximise the ratio between performance and effort, while considering the adjacency of structures that may have already been labelled – which decrease the length of the boundary to trace. We provide quantitative results on synthetically deformed MRI scans and real histological data, showing that our framework can reduce labelling effort by up to 60–70% without compromising accuracy

MENTORING DEEP LEARNING MODELS FOR MASS SCREENING WITH LIMITED DATA

Deep Learning (DL) has an extensively rich state-of-the-art literature in medical imaging analysis. However, it requires large amount of data to begin training. This limits its usage in tackling future epidemics, as one might need to wait for months and even years to collect fully annotated data, raising a fundamental question: is it possible to deploy AI-driven tool earlier in epidemics to mass screen the infected cases? For such a context, human/Expert in the loop Machine Learning (ML), or Active Learning (AL), becomes imperative enabling machines to commence learning from the first day with minimum available labeled dataset. In an unsupervised learning, we develop pretrained DL models that autonomously refine themselves through iterative learning, with human experts intervening only when the model misclassifies and for a limited amount of data. We introduce a new terminology for this process, calling it mentoring. We validated this concept in the context of Covid-19 in three distinct datasets: Chest X-rays, Computed Tomography (CT) scans, and cough sounds, each consisting of 1364, 4714, and 10,000 images, respectively. The framework classifies the deep features of the data into two clusters (0/1: Covid-19/non-Covid-19). Our main goal is to strongly emphasize the potential use of AL in predicting diseases during future epidemics. With this framework, we achieved the AUC scores of 0.76, 0.99, and 0.94 on cough sound, Chest X-rays, and CT scans dataset using only 40%, 33%, and 30% of the annotated dataset, respectively. For reproducibility, the link of implementation is provided: https://github.com/2ailab/Active-Learning

Reliability-based cleaning of noisy training labels with inductive conformal prediction in multi-modal biomedical data mining

Accurately labeling biomedical data presents a challenge. Traditional semi-supervised learning methods often under-utilize available unlabeled data. To address this, we propose a novel reliability-based training data cleaning method employing inductive conformal prediction (ICP). This method capitalizes on a small set of accurately labeled training data and leverages ICP-calculated reliability metrics to rectify mislabeled data and outliers within vast quantities of noisy training data. The efficacy of the method is validated across three classification tasks within distinct modalities: filtering drug-induced-liver-injury (DILI) literature with title and abstract, predicting ICU admission of COVID-19 patients through CT radiomics and electronic health records, and subtyping breast cancer using RNA-sequencing data. Varying levels of noise to the training labels were introduced through label permutation. Results show significant enhancements in classification performance: accuracy enhancement in 86 out of 96 DILI experiments (up to 11.4%), AUROC and AUPRC enhancements in all 48 COVID-19 experiments (up to 23.8% and 69.8%), and accuracy and macro-average F1 score improvements in 47 out of 48 RNA-sequencing experiments (up to 74.6% and 89.0%). Our method offers the potential to substantially boost classification performance in multi-modal biomedical machine learning tasks. Importantly, it accomplishes this without necessitating an excessive volume of meticulously curated training data

Explainable Active Learning for Preference Elicitation

Gaining insights into the preferences of new users and subsequently personalizing recommendations necessitate managing user interactions intelligently, namely, posing pertinent questions to elicit valuable information effectively. In this study, our focus is on a specific scenario of the cold-start problem, where the recommendation system lacks adequate user presence or access to other users' data is restricted, obstructing employing user profiling methods utilizing existing data in the system. We employ Active Learning (AL) to solve the addressed problem with the objective of maximizing information acquisition with minimal user effort. AL operates for selecting informative data from a large unlabeled set to inquire an oracle to label them and eventually updating a machine learning (ML) model. We operate AL in an integrated process of unsupervised, semi-supervised, and supervised ML within an explanatory preference elicitation process. It harvests user feedback (given for the system's explanations on the presented items) over informative samples to update an underlying ML model estimating user preferences. The designed user interaction facilitates personalizing the system by incorporating user feedback into the ML model and also enhances user trust by refining the system's explanations on recommendations. We implement the proposed preference elicitation methodology for food recommendation. We conducted human experiments to assess its efficacy in the short term and also experimented with several AL strategies over synthetic user profiles that we created for two food datasets, aiming for long-term performance analysis. The experimental results demonstrate the efficiency of the proposed preference elicitation with limited user-labeled data while also enhancing user trust through accurate explanations.Comment: Preprin

Empirical Evaluation of Semi-Supervised Naïve Bayes for Active Learning

This thesis describes an empirical evaluation of semi-supervised and active learning individually, and in combination for the naïve Bayes classifier. Active learning aims to minimise the amount of labelled data required to train the classifier by using the model to direct the labelling of the most informative unlabelled examples. The key difficulty with active learning is that the initial model often gives a poor direction for labelling the unlabelled data in the early stages. However, using both labelled and unlabelled data with semi-supervised learning might be achieve a better initial model because the limited labelled data are augmented by the information in the unlabelled data. In this thesis, a suite of benchmark datasets is used to evaluate the benefit of semi-supervised learning and presents the learning curves for experiments to compare the performance of each approach. First, we will show that the semi-supervised naïve Bayes does not significantly improve the performance of the naïve Bayes classifier. Subsequently, a down-weighting technique is used to control the influence of the unlabelled data, but again this does not improve performance. In the next experiment, a novel algorithm is proposed by using a sigmoid transformation to recalibrate the overly confident naïve Bayes classifier. This algorithm does not significantly improve on the naïve Bayes classifier, but at least does improve the semi-supervised naïve Bayes classifier. In the final experiment we investigate the effectiveness of the combination of active and semi-supervised learning and empirically illustrate when the combination does work, and when does not

Process monitoring and machine learning for defect detection in laser-based metal additive manufacturing

Over the past several decades, metal Additive Manufacturing (AM) has transitioned from a rapid prototyping method to a viable manufacturing tool. AM technologies can produce parts on-demand, repair damaged components, and provide an increased freedom of design not previously attainable by traditional manufacturing techniques. The increasing maturation of metal AM is attracting high-value industries to directly produce components for use in aerospace, automotive, biomedical, and energy fields. Two leading processes for metal part production are Powder Bed Fusion with laser beam (PBF-LB/M) and Directed Energy Deposition with laser beam (DED-LB/M). Despite the many advances made with these technologies, the highly dynamic nature of the process frequently results in the formation of defects. These technologies are also notoriously difficult to control, and the existing machines do not offer closed loop control. In the present work, the application of various Machine Learning (ML) approaches and in-situ monitoring technologies for the purpose of defect detection are reviewed. The potential of these methods for enabling process control implementation is discussed. We provide a critical review of trends in the usage of data structures and ML algorithms and compare the capabilities of different sensing technologies and their application to monitoring tasks in laser metal AM. The future direction of this field is then discussed, and recommendations for further research are provided