Schmettow, Martin

Schraagen, Johannes Martinus Cornelis

van Merkerk, Rutger

English

University of Twente Research Information

User interface issues with infusion pumps: A systematic review and guidelines for usability testing Jan Maarten Schraagen1, Ph.D., Martin Schmettow2, Ph.D., & Rutger van Merkerk3, Ph.D.    1TNO & University of Twente; 2University of Twente; 3 University Medical Center Utrecht Overview ‘Patient Safety’ Project  Systematic literature review  Issues in Human Factors Validation Testing: Number of test participants required Population of intended users  Empirical test to address issues  Conclusions and recommendations  Baltimore, March 12, 2012 HFE in Health Care Powered by Partnership UMC Utrecht Exploitatie B.V. Patient Safety Project  Three-year project (2011-2013), aimed at improving patient safety through two main tracks:  Improving human-technology interaction  Improving human-human interaction   Improving human-technology interaction:  Cooperation between academic teaching hospital, university, research organisation, small business  Focus on improving interfaces for infusion pumps  Approach: literature review, user requirements elicitation, prioritizing user requirements through interviews, iterative interface design development, usability testing  Systematic literature review1 Review (N=55) showed wide variety of usability issues, but also many non-usability issues (e.g., issues during pump procurement process)  Many opportunities where things can go wrong during the infusion process, particularly considering widespread pump usage  76 requirements were derived from the literature, grouped in 9 use cases (e.g., placement/removal syringe; administer bolus; start/stop infusion)  Interviews with total of 7 ‘super-users’ from 3 departments (OR, ICU, Nursing) led to prioritizing of requirements and confirmed validity Baltimore, March 12, 2012 HFE in Health Care 1 Schraagen et al. (under review). User-interface issues with infusion pumps: A systematic review. Submitted to J Biomed Informatics  From requirements to testing: Issues in Human Factors Validation Testing In our interviews, we noted differences in priorities among departments     Medical devices constitute high-risk equipment, for which established standards1 for estimating sample sizes in usability testing may be inappropriate    1 e.g., Virzi, 1992; Nielsen, 1993, Lewis, 2001; Faulkner, 2003 Baltimore, March 12, 2012 HFE in Health Care Issue #1: Generalization regarding safe and effective use                 by the ultimate population of users Issue #2: Evidence-based determination of sample size in                  high-risk equipment  Current quantitative control strategies for use in evaluation studies Baltimore, March 12, 2012 HFE in Health Care Set goal for detection rate D Set number of users to 5 Run study Run study Estimate number of users Run a few sessions Set goal for detection rate D Terminate study Estimate # of remaining defects Run session Set goal for detection rate DA priori control (Nielsen) Early control (Lewis) Late control (Schmettow) Goal? No Yes Limitations of previous approaches1 Virzi’s formula underestimates remaining number of defects, due to variance in defect visibility  Medical devices may differ from software products in problem detection rates  Goals for detection rate D may need to be set higher for medical devices than ~80%-85%, leading to higher numbers of users to be tested than the standard five recommended by Nielsen  Baltimore, March 12, 2012 HFE in Health Care 1Schmettow, M. (2012). Sample Sizes for Usability Studies. Communications of the ACM, 55(4) Current study1 Goal: to evaluate a late control strategy with a medical device for different user groups  High-level method: Develop novel interface design for infusion pump Select representative user groups (OR and ICU) Select representative tasks for users to carry out with infusion pump Observe user problems and apply ‘triage strategy’ (sanitize dataset) Apply late control strategy: Set goal D = 90%, with 90% CI Run session with subsample and estimate # remaining defects Continue until goal D = 90% is reached Baltimore, March 12, 2012 HFE in Health Care 1 Vos, W.M. (2011). Quantitative and efficient usability testing in high risk system development. Unpublished Master’s thesis. Novel interface design, presented on touch- screen Baltimore, March 12, 2012 HFE in Health Care User groups   Level of education (distribution) MABAVocationalBaltimore, March 12, 2012 HFE in Health Care Age (distribution) 20-2930-3940-4950-59OR anaesthesiologists: N=18 ICU nurses: N=18 2 participants excluded due to incomplete video data 051015OR ICU totalaveragePump experience (years) Tasks Fixed set of 11 tasks covering main functions of infusion pump  Typical tasks: interpreting the meaning of an alarm, adjusting values, and checking pump status  Tasks were piloted with three experts and were assessed as being: Externally valid Of roughly equal difficulty Independent of each other Baltimore, March 12, 2012 HFE in Health Care Procedure Setting: controlled, quiet environment in 1,042-bed academic teaching hospital Consent form and Pre-task demographics questionnaire Think-aloud while performing 11 tasks (video and audio recorded as well as screen captures) After each task: immediate retrospective think aloud (1 minute) After all 11 tasks were performed, three post-task questionnaires: 1. 72-item design features questionnaire (5-point Likert scale) 2. 2-item semantic differential scale on CTA experience 3. Exterior appearance semantic differential scales  Full procedure completed within 90 minutes Baltimore, March 12, 2012 HFE in Health Care Triage Expert judgment Think-aloud Questionnaire Baltimore, March 12, 2012 HFE in Health Care Definitely not a usability problem Undecided Definitely a usability problem “Full (‘raw’) data set” “Stripped data set” Triage Expert judgment Think-aloud Questionnaire Baltimore, March 12, 2012 HFE in Health Care Definitely not a usability problem Undecided Definitely a usability problem Score CTA Score Quest Score   Expert Combi  3 1 1 1 Definitely not a usability problem --> observed, not reported by subjects in Quest, expert not a problem 3 2 1 1 Definitely not a usability problem --> observed, reported by subjects in Quest, expert opinion not a problem 3 1 2 2 Undecided --> observed, not reported by subjects in Quest, Expert unsure 3 2 2 2 Undecided --> observed, reported by subjects, unsure by expert 2 1 1 1 Definitely not a usability problem --> utterances during performance, not reported by subjects in Quest, Expert opinion not a problem 2 2 1 1 Definitely not a usability problem --> utterances during performance, reported by subjects in Quest, Expert opinion not a problem 2 1 2 2 Undecided --> utterances during performance, not reported by subjects in Quest, Expert opinion unsure 2 2 2 2 Undecided --> utterances during performing, reported by subjects in Quest, expert opinion unsure  Ergonomical and cognitive design principles (Rubin, 1994) Results Baltimore, March 12, 2012 HFE in Health Care • N=10 (OR) • N=10 (ICU) Phase 1 • N=7 (OR) • N=7 (ICU) Phase 2 109 (89) problems observed 86 (75) problems observed Example progress analysis Baltimore, March 12, 2012 HFE in Health Care *First phase ICU-trials (N=10) Process figures Progress figures (%D) < 85% Baltimore, March 12, 2012 HFE in Health Care Quantitative results > 85%  more efficiënt data set  AnPh2  D = 46%  AnPh2  D = 31% Contribution problems detected once Baltimore, March 12, 2012 HFE in Health Care Aantal keer opgemerkt (n)    Large contribution of problems detected once in the full data set compared to stripped data set Problem distribution between anesthesiologists and ICU-nurses  Baltimore, March 12, 2012 HFE in Health Care  Some problems are observed (more often) by either of the user groups     Conclusions On the basis of the raw data set, the goal of 90% detection was never reached, not even with the combined sample of N=34  Goal of 90% detection was only reached when the data set was stripped of problems that were definitely not a usability problem AND when the two user groups were combined (N=34) Extrapolation under these assumptions to D=95% leads to N=66, and for D=98% to N=129  Model-based predictions of problems detected are highly sensitive to: Individual differences in experience levels Problems mentioned only once User groups   Baltimore, March 12, 2012 HFE in Health Care Number of users required for each major user group, for goal D=90% (stripped data set) Baltimore, March 12, 2012 HFE in Health Care N=31 OR users 90% CI: .78-.95 N=21 ICU users 90% CI: .80-.97 Recommendations Do not use the magic number approach, use the late-control strategy instead Pay more attention to the quality of the data set, and use triage-like methods to sanitize the data set  Variance in defect visibility exists and may lead to gross underestimates of the number of users required, particularly when different user groups need to be taken into account Baltimore, March 12, 2012 HFE in Health Care 

User interface issues with infusion pumps: A systematic review and guidelines for usability testing

Abstract

Similar works

Full text

Available Versions

University of Twente Research Information

NARCIS