Guidance can support users during the exploration and analysis of complex
data. Previous research focused on characterizing the theoretical aspects of
guidance in visual analytics and implementing guidance in different scenarios.
However, the evaluation of guidance-enhanced visual analytics solutions remains
an open research question. We tackle this question by introducing and
validating a practical evaluation methodology for guidance in visual analytics.
We identify eight quality criteria to be fulfilled and collect expert feedback
on their validity. To facilitate actual evaluation studies, we derive two sets
of heuristics. The first set targets heuristic evaluations conducted by expert
evaluators. The second set facilitates end-user studies where participants
actually use a guidance-enhanced system. By following such a dual approach, the
different quality criteria of guidance can be examined from two different
perspectives, enhancing the overall value of evaluation studies. To test the
practical utility of our methodology, we employ it in two studies to gain
insight into the quality of two guidance-enhanced visual analytics solutions,
one being a work-in-progress research prototype, and the other being a publicly
available visualization recommender system. Based on these two evaluations, we
derive good practices for conducting evaluations of guidance in visual
analytics and identify pitfalls to be avoided during such studies.Comment: Accepted to IEEE VIS 202