Story generation and understanding -- as with all NLG/NLU tasks -- has seen a
surge in neurosymbolic work. Researchers have recognized that, while large
language models (LLMs) have tremendous utility, they can be augmented with
symbolic means to be even better and to make up for any flaws that the neural
networks might have. However, symbolic methods are extremely costly in terms of
the amount of time and expertise needed to create them. In this work, we
capitalize on state-of-the-art Code-LLMs, such as Codex, to bootstrap the use
of symbolic methods for tracking the state of stories and aiding in story
understanding. We show that our CoRRPUS system and abstracted prompting
procedures can beat current state-of-the-art structured LLM techniques on
pre-existing story understanding tasks (bAbI task 2 and Re^3) with minimal hand
engineering. We hope that this work can help highlight the importance of
symbolic representations and specialized prompting for LLMs as these models
require some guidance for performing reasoning tasks properly.Comment: Accepted to Findings of ACL 202