FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning

Pseudo labeling and consistency regularization approaches based on confidencethresholding have made great progress in semi-supervised learning (SSL).However, we argue that existing methods might fail to adopt suitable thresholdssince they either use a pre-defined / fixed threshold or an ad-hoc thresholdadjusting scheme, resulting in inferior performance and slow convergence. Wefirst analyze a motivating example to achieve some intuitions on therelationship between the desirable threshold and model's learning status. Basedon the analysis, we hence propose FreeMatch to define and adjust the confidencethreshold in a self-adaptive manner according to the model's learning status.We further introduce a self-adaptive class fairness regularization penalty thatencourages the model to produce diverse predictions during the early stages oftraining. Extensive experimental results indicate the superiority of FreeMatchespecially when the labeled data are extremely rare. FreeMatch achieves 5.78%,13.59%, and 1.28% error rate reduction over the latest state-of-the-art methodFlexMatch on CIFAR-10 with 1 label per class, STL-10 with 4 labels per class,and ImageNet with 100 labels per class, respectively.<br

Dropout Training as Adaptive Regularization

Dropout and other feature noising schemes control overfitting by artificially corrupting the training data. For generalized linear models, dropout performs a form of adaptive regularization. Using this viewpoint, we show that the dropout regularizer is first-order equivalent to an L2 regularizer applied after scaling the features by an estimate of the inverse diagonal Fisher information matrix. We also establish a connection to AdaGrad, an online learning algorithm, and find that a close relative of AdaGrad operates by repeatedly solving linear dropout-regularized problems. By casting dropout as regularization, we develop a natural semi-supervised algorithm that uses unlabeled data to create a better adaptive regularizer. We apply this idea to document classification tasks, and show that it consistently boosts the performance of dropout training, improving on state-of-the-art results on the IMDB reviews dataset.Comment: 11 pages. Advances in Neural Information Processing Systems (NIPS), 201

Unsupervised Domain Adaptation: A Multi-task Learning-based Method

This paper presents a novel multi-task learning-based method for unsupervised domain adaptation. Specifically, the source and target domain classifiers are jointly learned by considering the geometry of target domain and the divergence between the source and target domains based on the concept of multi-task learning. Two novel algorithms are proposed upon the method using Regularized Least Squares and Support Vector Machines respectively. Experiments on both synthetic and real world cross domain recognition tasks have shown that the proposed methods outperform several state-of-the-art domain adaptation methods

Domain adaptive segmentation in volume electron microscopy imaging

In the last years, automated segmentation has become a necessary tool for volume electron microscopy (EM) imaging. So far, the best performing techniques have been largely based on fully supervised encoder-decoder CNNs, requiring a substantial amount of annotated images. Domain Adaptation (DA) aims to alleviate the annotation burden by 'adapting' the networks trained on existing groundtruth data (source domain) to work on a different (target) domain with as little additional annotation as possible. Most DA research is focused on the classification task, whereas volume EM segmentation remains rather unexplored. In this work, we extend recently proposed classification DA techniques to an encoder-decoder layout and propose a novel method that adds a reconstruction decoder to the classical encoder-decoder segmentation in order to align source and target encoder features. The method has been validated on the task of segmenting mitochondria in EM volumes. We have performed DA from brain EM images to HeLa cells and from isotropic FIB/SEM volumes to anisotropic TEM volumes. In all cases, the proposed method has outperformed the extended classification DA techniques and the finetuning baseline. An implementation of our work can be found on https://github.com/JorisRoels/domain-adaptive-segmentation