Understanding and Supporting Debugging Workflows in Multiverse Analysis

Abstract

Multiverse analysis-a paradigm for statistical analysis that considers all combinations of reasonable analysis choices in parallel-promises to improve transparency and reproducibility. Although recent tools help analysts specify multiverse analyses, they remain difficult to use in practice. In this work, we conduct a formative study with four multiverse researchers, which identifies debugging as a key barrier. We find debugging is challenging because of the latency between running analyses and detecting bugs, and the scale of metadata needed to be processed to diagnose a bug. To address these challenges, we prototype a command-line interface tool, Multiverse Debugger, which helps diagnose bugs in the multiverse and propagate fixes. In a second, focused study (n=13), we use Multiverse Debugger as a probe to develop a model of debugging workflows and identify challenges, including the difficulty in understanding the composition of a multiverse. We conclude with design implications for future multiverse analysis authoring systems