The evaluator effect: A chilling fact about usability evaluation methods

Computer professionals have a need for robust, easy-to-use usability evaluation methods (UEMs) to help them systematically improve the usability of computer arti-facts. However, cognitive walkthrough (CW), heuristic evaluation (HE), and thinking-aloud study (TA)—3 of the most widely used UEMs—suffer from a substantial evalua-tor effect in that multiple evaluators evaluating the same interface with the same UEM detect markedly different sets of problems. A review of 11 studies of these 3 UEMs re-veals that the evaluator effect exists for both novice and experienced evaluators, for both cosmetic and severe problems, for both problem detection and severity assess-ment, and for evaluations of both simple and complex systems. The average agree-ment between any 2 evaluators who have evaluated the same system using the same UEM ranges from 5 % to 65%, and no 1 of the 3 UEMs is consistently better than the others. Although evaluator effects of this magnitude may not be surprising for a UEM as informal as HE, it is certainly notable that a substantial evaluator effect persists for evaluators who apply the strict procedure of CW or observe users thinking out loud. Hence, it is highly questionable to use a TA with 1 evaluator as an authoritative state-ment about what problems an interface contains. Generally, the application of the UEMs is characterized by (a) vague goal analyses leading to variability in the task sce-narios, (b) vague evaluation procedures leading to anchoring, or (c) vague problem criteria leading to anything being accepted as a usability problem, or all of these. Th

The evaluator effect during first-time use of the cognitive walkthrough technique

Practising system developers without a human factors background need robust, easy-to-use usability evaluation methods. The cognitive walkthrough (CW) technique (Lewis et al. 1990, Wharton et al. 1994) has been devised to provide such a method and is particularly suited to evaluate designs before testing wit

A Global Perspective: Current Trends and Issues in ICT for 21st Century Education.

Around the world educators, policy makers, and others share a universal goal to ensure that all learners reach their full potential; we seek best practices to prepare educators and leaders to improve student learning, prepare learners for their futures, maintain current educators’ knowledge and skills, increase student engagement, and integrate learning technologies into their curriculum. Questions have been raised regarding the role of practicum, appropriate curriculum, and use of technology in this preparation and for ongoing professional development. The discovery and sharing of a purposeful and systemic plan will require many educators to work together, share lessons learned, and invest energy in promoting policies to bring about changes. This paper brings together updates from experts from around the globe; each contributor has extensive experience and knowledge to share from his/her country’s perspective. The purpose of this paper is to share what these individuals know from their experience, and to generate a discussion within our educational community on questions of future research, learning from each other, and identifying opportunities for collaboration