QuaLA-MiniLM: a Quantized Length Adaptive MiniLM

Limited computational budgets often prevent transformers from being used in production and from having their high accuracy utilized. A knowledge distillation approach addresses the computational efficiency by self-distilling BERT into a smaller transformer representation having fewer layers and smaller internal embedding. However, the performance of these models drops as we reduce the number of layers, notably in advanced NLP tasks such as span question answering. In addition, a separate model must be trained for each inference scenario with its distinct computational budget. Dynamic-TinyBERT tackles both limitations by partially implementing the Length Adaptive Transformer (LAT) technique onto TinyBERT, achieving x3 speedup over BERT-base with minimal accuracy loss. In this work, we expand the Dynamic-TinyBERT approach to generate a much more highly efficient model. We use MiniLM distillation jointly with the LAT method, and we further enhance the efficiency by applying low-bit quantization. Our quantized length-adaptive MiniLM model (QuaLA-MiniLM) is trained only once, dynamically fits any inference scenario, and achieves an accuracy-efficiency trade-off superior to any other efficient approaches per any computational budget on the SQuAD1.1 dataset (up to x8.8 speedup with <1% accuracy loss). The code to reproduce this work is publicly available on Github.Comment: In this version we added reference to the source code in the abstract. arXiv admin note: text overlap with arXiv:2111.0964

The Spin Structure of the Nucleon

We present an overview of recent experimental and theoretical advances in our understanding of the spin structure of protons and neutrons.Comment: 84 pages, 29 figure

The Academic Library as a Teaching Library: A Role for the 1980s

published or submitted for publicatio

An Antioch career : the memoirs of J. Dudley Dawson

Biography of J. Dudley Dawson, former president of Antioch College.https://aura.antioch.edu/facbooks/1014/thumbnail.jp

Notes from a pragmatic idealist: Selected papers, 1985-1997

Series of essays and speeches by the former president and chancellor of Antioch Universityhttps://aura.antioch.edu/facbooks/1085/thumbnail.jp