Despite recent success in large language model (LLM) reasoning, LLMs struggle
with hierarchical multi-step reasoning tasks like generating complex programs.
For these tasks, humans often start with a high-level algorithmic design and
implement each part gradually. We introduce Parsel, a framework enabling
automatic implementation and validation of complex algorithms with code LLMs,
taking hierarchical function descriptions in natural language as input. We show
that Parsel can be used across domains requiring hierarchical reasoning,
including program synthesis, robotic planning, and theorem proving. We show
that LLMs generating Parsel solve more competition-level problems in the APPS
dataset, resulting in pass rates that are over 75% higher than prior results
from directly sampling AlphaCode and Codex, while often using a smaller sample
budget. We also find that LLM-generated robotic plans using Parsel as an
intermediate language are more than twice as likely to be considered accurate
than directly generated plans. Lastly, we explore how Parsel addresses LLM
limitations and discuss how Parsel may be useful for human programmers.Comment: new quantitative detail