Pre-trained language models (PLMs) have achieved remarkable results on NLP
tasks but at the expense of huge parameter sizes and the consequent
computational costs. In this paper, we propose Variator, a parameter-efficient
acceleration method that enhances computational efficiency through
plug-and-play compression plugins. Compression plugins are designed to reduce
the sequence length via compressing multiple hidden vectors into one and
trained with original PLMs frozen. Different from traditional model
acceleration methods, which compress PLMs to smaller sizes, Variator offers two
distinct advantages: (1) In real-world applications, the plug-and-play nature
of our compression plugins enables dynamic selection of different compression
plugins with varying acceleration ratios based on the current workload. (2) The
compression plugin comprises a few compact neural network layers with minimal
parameters, significantly saving storage and memory overhead, particularly in
scenarios with a growing number of tasks. We validate the effectiveness of
Variator on seven datasets. Experimental results show that Variator can save
53% computational costs using only 0.9% additional parameters with a
performance drop of less than 2%. Moreover, when the model scales to billions
of parameters, Variator matches the strong performance of uncompressed PLMs.Comment: Accepted by Findings of EMNL