1 research outputs found
Paths Explored, Paths Omitted, Paths Obscured: Decision Points & Selective Reporting in End-to-End Data Analysis
Drawing reliable inferences from data involves many, sometimes arbitrary,
decisions across phases of data collection, wrangling, and modeling. As
different choices can lead to diverging conclusions, understanding how
researchers make analytic decisions is important for supporting robust and
replicable analysis. In this study, we pore over nine published research
studies and conduct semi-structured interviews with their authors. We observe
that researchers often base their decisions on methodological or theoretical
concerns, but subject to constraints arising from the data, expertise, or
perceived interpretability. We confirm that researchers may experiment with
choices in search of desirable results, but also identify other reasons why
researchers explore alternatives yet omit findings. In concert with our
interviews, we also contribute visualizations for communicating decision
processes throughout an analysis. Based on our results, we identify design
opportunities for strengthening end-to-end analysis, for instance via tracking
and meta-analysis of multiple decision paths