5 research outputs found
BatchPrompt: Accomplish more with less
As the ever-increasing token limits of large language models (LLMs) have
enabled long context as input, prompting with single data samples might no
longer an efficient way. A straightforward strategy improving efficiency is to
batch data within the token limit (e.g., 8k for gpt-3.5-turbo; 32k for GPT-4),
which we call BatchPrompt. We have two initial observations for prompting with
batched data. First, we find that prompting with batched data in longer
contexts will inevitably lead to worse performance, compared to single-data
prompting. Second, the performance of the language model is significantly
correlated with the positions and order of the batched data, due to the
corresponding change in decoder context. To retain efficiency and overcome
performance loss, we propose Batch Permutation and Ensembling (BPE), and a
novel Self-reflection-guided EArly Stopping (SEAS) technique. Our comprehensive
experimental evaluation demonstrates that BPE can boost the performance of
BatchPrompt with a striking margin on a range of popular NLP tasks, including
question answering (Boolq), textual entailment (RTE), and duplicate questions
identification (QQP). These performances are even competitive with/higher than
single-data prompting(SinglePrompt), while BatchPrompt requires much fewer LLM
calls and input tokens (For SinglePrompt v.s. BatchPrompt with batch size 32,
using just 9%-16% the number of LLM calls, Boolq accuracy 90.6% to 90.9% with
27.4% tokens, QQP accuracy 87.2% to 88.4% with 18.6% tokens, RTE accuracy 91.5%
to 91.1% with 30.8% tokens). To the best of our knowledge, this is the first
work to technically improve prompting efficiency of large language models. We
hope our simple yet effective approach will shed light on the future research
of large language models. The code will be released.Comment: 20 pages, 5 figure
Recommended from our members
Distribution distance measures in generative and privacy models
Distribution distance measures provide a useful class of tools for generative and privacy models. In both cases, the goal is to simulate a data distribution without revealing too much about individual points. While early generative models focused on matching data in a component-wise manner, the models in this work incorporate distribution metrics to provide population-level information during training. Doing so reduces overfitting and increases the model's ability to generalize. Maximum mean discrepancy and energy distance are two such metrics that are easily defined and implemented over samples, and provide meaningful results on a range of data sets and data types. This work presents three main contributions: (1) a novel use of importance weights to modify the output distribution of a generative model, (2) an application and evaluation of a generative model for medical data privacy, and (3) a novel method for private data synthesis using support points and differential privacyStatistic