The scale of large pre-trained models (PTMs) poses significant challenges in
adapting to downstream tasks due to the high optimization overhead and storage
costs associated with full-parameter fine-tuning. To address this, many studies
explore parameter-efficient tuning methods, also framed as "delta tuning",
which updates only a small subset of parameters, known as "delta modules",
while keeping the backbone model's parameters fixed. However, the practicality
and flexibility of delta tuning have been limited due to existing
implementations that directly modify the code of the backbone PTMs and
hard-code specific delta tuning methods for each PTM. In this paper, we present
OpenDelta, an open-source library that overcomes these limitations by providing
a plug-and-play implementation of various delta tuning methods. Our novel
techniques eliminate the need to modify the backbone PTMs' code, making
OpenDelta compatible with different, even novel PTMs. OpenDelta is designed to
be simple, modular, and extensible, providing a comprehensive platform for
researchers and practitioners to adapt large PTMs efficiently.Comment: Accepted to ACL 2023 Demo trac