The success of large language models (LLMs), like GPT-4 and ChatGPT, has led
to the development of numerous cost-effective and accessible alternatives that
are created by finetuning open-access LLMs with task-specific data (e.g.,
ChatDoctor) or instruction data (e.g., Alpaca). Among the various fine-tuning
methods, adapter-based parameter-efficient fine-tuning (PEFT) is undoubtedly
one of the most attractive topics, as it only requires fine-tuning a few
external parameters instead of the entire LLMs while achieving comparable or
even better performance. To enable further research on PEFT methods of LLMs,
this paper presents LLM-Adapters, an easy-to-use framework that integrates
various adapters into LLMs and can execute these adapter-based PEFT methods of
LLMs for different tasks. The framework includes state-of-the-art open-access
LLMs such as LLaMA, BLOOM, and GPT-J, as well as widely used adapters such as
Series adapters, Parallel adapter, Prompt-based learning and
Reparametrization-based methods. Moreover, we conduct extensive empirical
studies on the impact of adapter types, placement locations, and
hyper-parameters to the best design for each adapter-based methods. We evaluate
the effectiveness of the adapters on fourteen datasets from two different
reasoning tasks, Arithmetic Reasoning and Commonsense Reasoning. The results
demonstrate that using adapter-based PEFT in smaller-scale LLMs (7B) with few
extra trainable parameters yields comparable, and in some cases superior,
performance to powerful LLMs (175B) in zero-shot inference on both reasoning
tasks.Comment: EMNLP 2023. The code of our framework can be found at
https://github.com/AGI-Edgerunners/LLM-Adapters. We will keep all of the code
open-source and continue to update the framework with new adapters, LLMs, and
task