Facilitating heuristic evaluation for novice evaluators

Heuristic evaluation (HE) is one of the most widely used usability evaluation methods. The reason for its popularity is that it is a discount method, meaning that it does not require substantial time or resources, and it is simple, as evaluators can evaluate a system guided by a set of usability heuristics. Despite its simplicity, a major problem with HE is that there is a significant gap in the quality of results produced by expert and novice evaluators. This gap has made some scholars question the usefulness of the method as they claim that the evaluation results are a product of the evaluator’s experience rather than the method itself. In response, the goal of this thesis is to bridge the gap between expert and novice evaluators. Based on interviews with 15 usability experts, which focused on their experience with the method, the difficulties they faced when they were novices, and how they overcame such difficulties, it presents a comprehensive protocol called Coherent Heuristic Evaluation (CoHE). This step-by-step protocol guides novice evaluators from the moment they decide to conduct an evaluation until the submission of their evaluation report. This protocol was verified by conducting an experiment to observe the difference between novices using the CoHE protocol and novices using Nielsen’s 10 usability heuristics without the guidance. The experiment involved 20 novices performing two sessions; the first was an understanding session where the novices read and understood the heuristics and the second was an inspecting session where they inspected a system. The findings show that, while evaluators take more time to read and evaluate a system using CoHE, they tend to identify more problems. The experiment also demonstrated that CoHE can improve the thoroughness, effectiveness, and f-measure of evaluation. However, the validity of CoHE was comparable to that of HE