In contrast to objectively measurable aspects (such as accuracy, reading
speed, or memorability), the subjective experience of visualizations has only
recently gained importance, and we have less experience how to measure it. We
explore how subjective experience is affected by chart design using multiple
experimental methods. We measure the effects of changes in color, orientation,
and source annotation on the perceived readability and trustworthiness of
simple bar charts. Three different experimental designs (single image rating,
forced choice comparison, and semi-structured interviews) provide similar but
different results. We find that these subjective experiences are different from
what prior work on objective dimensions would predict. Seemingly
inconsequential choices, like orientation, have large effects for some methods,
indicating that study design alters decision-making strategies. Next to
insights into the effect of chart design, we provide methodological insights,
such as a suggested need to carefully isolate individual elements in charts to
study subjective experiences.Comment: 19 pages, 5 figures, 2 table