Fine-tuning BERT-based models is resource-intensive in memory, computation,
and time. While many prior works aim to improve inference efficiency via
compression techniques, e.g., pruning, these works do not explicitly address
the computational challenges of training to downstream tasks. We introduce
Learner modules and priming, novel methods for fine-tuning that exploit the
overparameterization of pre-trained language models to gain benefits in
convergence speed and resource utilization. Learner modules navigate the double
bind of 1) training efficiently by fine-tuning a subset of parameters, and 2)
training effectively by ensuring quick convergence and high metric scores. Our
results on DistilBERT demonstrate that learners perform on par with or surpass
the baselines. Learners train 7x fewer parameters than state-of-the-art methods
on GLUE. On CoLA, learners fine-tune 20% faster, and have significantly lower
resource utilization.Comment: 8 pages, 9 figures, 2 tables, presented at ICML 2022 workshop on
Hardware-Aware Efficient Training (HAET 2022