Additionally, the strong dependency among in-context examples makes it an
NP-hard combinatorial optimization problem and enumerating all permutations is
infeasible. Hence we propose LENS, a fiLter-thEN-Search method to tackle this
challenge in two stages: First we filter the dataset to obtain informative
in-context examples individually. Specifically, we propose a novel metric,
InfoScore, to evaluate the example's in-context informativeness based on the
language model's feedback, and further propose a progressive filtering process
to filter out uninformative examples. Then we propose diversity-guided example
search which iteratively refines and evaluates the selected example
permutations, to find examples that fully depict the task. The experimental
results show that LENS significantly outperforms a wide range of baselines.Comment: Accepted to the Findings of EMNLP 202