Chain-of-Thought (CoT) prompting can dramatically improve the multi-step
reasoning abilities of large language models (LLMs). CoT explicitly encourages
the LLM to generate intermediate rationales for solving a problem, by providing
a series of reasoning steps in the demonstrations. Despite its success, there
is still little understanding of what makes CoT prompting effective and which
aspects of the demonstrated reasoning steps contribute to its performance. In
this paper, we show that CoT reasoning is possible even with invalid
demonstrations - prompting with invalid reasoning steps can achieve over 80-90%
of the performance obtained using CoT under various metrics, while still
generating coherent lines of reasoning during inference. Further experiments
show that other aspects of the rationales, such as being relevant to the query
and correctly ordering the reasoning steps, are much more important for
effective CoT reasoning. Overall, these findings both deepen our understanding
of CoT prompting, and open up new questions regarding LLMs' capability to learn
to reason in context.Comment: ACL-23 Camera Ready. Code and model input/output are available at
https://github.com/sunlab-osu/Understanding-Co