Large language models (LLMs) have exhibited remarkable capabilities in
learning from explanations in prompts. Yet, there has been limited
understanding of what makes explanations effective for in-context learning.
This work aims to better understand the mechanisms by which explanations are
used for in-context learning. We first study the impact of two different
factors on prompting performance when using explanations: the computation trace
(the way the solution is decomposed) and the natural language of the prompt. By
perturbing explanations on three controlled tasks, we show that both factors
contribute to the effectiveness of explanations, indicating that LLMs do
faithfully follow the explanations to some extent. We further study how to form
maximally effective sets of explanations for solving a given test query. We
find that LLMs can benefit from the complementarity of the explanation set as
they are able to fuse different reasoning specified by individual exemplars in
prompts. Additionally, having relevant exemplars also contributes to more
effective prompts. Therefore, we propose a maximal-marginal-relevance-based
exemplar selection approach for constructing exemplar sets that are both
relevant as well as complementary, which successfully improves the in-context
learning performance across three real-world tasks on multiple LLMs