Deep learning (DL) algorithms rely on massive amounts of labeled data.
Semi-supervised learning (SSL) and active learning (AL) aim to reduce this
label complexity by leveraging unlabeled data or carefully acquiring labels,
respectively. In this work, we primarily focus on designing an AL algorithm but
first argue for a change in how AL algorithms should be evaluated. Although
unlabeled data is readily available in pool-based AL, AL algorithms are usually
evaluated by measuring the increase in supervised learning (SL) performance at
consecutive acquisition steps. Because this measures performance gains from
both newly acquired instances and newly acquired labels, we propose to instead
evaluate the label efficiency of AL algorithms by measuring the increase in SSL
performance at consecutive acquisition steps. After surveying tools that can be
used to this end, we propose our neural pre-conditioning (NPC) algorithm
inspired by a Neural Tangent Kernel (NTK) analysis. Our algorithm incorporates
the classifier's uncertainty on unlabeled data and penalizes redundant samples
within candidate batches to efficiently acquire a diverse set of informative
labels. Furthermore, we prove that NPC improves downstream training in the
large-width regime in a manner previously observed to correlate with
generalization. Comparisons with other AL algorithms show that a
state-of-the-art SSL algorithm coupled with NPC can achieve high performance
using very few labeled data.Comment: NeurIPS 202