The use of think-aloud and instant data analysis in evaluation research: Exemplar and lessons learned

AbstractWhile health information technologies have become increasingly popular, many have not been formally tested to ascertain their usability. Traditional rigorous methods take significant amounts of time and manpower to evaluate the usability of a system. In this paper, we evaluate the use of instant data analysis (IDA) as developed by Kjeldskov et al. to perform usability testing on a tool designed for older adults and caregivers. The IDA method is attractive because it takes significantly less time and manpower than the traditional usability testing methods. In this paper we demonstrate how IDA was used to evaluate usability of a multifunctional wellness tool, discuss study results and lessons learned while using this method. We also present findings from an extension of the method which allows the grouping of similar usability problems in an efficient manner. We found that the IDA method is a quick, relatively easy approach to identifying and ranking usability issues among health information technologies

Safety and usability evaluation of a web-based insulin self-titration system for patients with type 2 diabetes mellitus

The rising incidence of type 2 diabetes mellitus (T2DM) induces severe challenges for the health care system. Our research group developed a web-based system named PANDIT that provides T2DM patients with insulin dosing advice using state of the art clinical decision support technology. The PANDIT interface resembles a glucose diary and provides advice through pop-up messages. Diabetes nurses (DNs) also have access to the system, allowing them to intervene when needed. The objective of this study was to establish whether T2DM patients can safely use PANDIT at home. To this end, we assessed whether patients experience usability problems with a high risk of compromising patient safety when interacting with the system, and whether PANDIT's insulin dosing advice are clinically safe. The study population consisted of patients with T2DM (aged 18-80) who used a once daily basal insulin as well as DNs from a university hospital. The usability evaluation consisted of think-aloud sessions with four patients and three DNs. Video data, audio data and verbal utterances were analyzed for usability problems encountered during PANDIT interactions. Usability problems were rated by a physician and a usability expert according to their potential impact on patient safety. The usability evaluation was followed by an implementation with a duration of four weeks. This implementation took place at the patients' homes with ten patients to evaluate clinical safety of PANDIT advice. PANDIT advice were systematically compared with DN advice. Deviating advice were evaluated with respect to patient safety by a panel of experienced physicians, which specialized in diabetes care. We detected seventeen unique usability problems, none of which was judged to have a high risk of compromising patient safety. Most usability problems concerned the lay-out of the diary, which did not clearly indicate which data entry fields had to be entered in order to obtain an advice. 27 out of 74 (36.5%) PANDIT advice differed from those provided by DNs. However, only one of these (1.4%) was considered unsafe by the panel. T2DM patients with no prior experience with the web-based self-management system were capable of consulting the system without encountering significant usability problems. Furthermore, the large majority of PANDIT advice were considered clinically safe according to the expert panel. One advice was considered unsafe. This could however easily be corrected by implementing a small modification to the system's knowledge bas

The role of various risk factors in the prevalence of cardiac autonomic neuropathy and associated diseases

The objectives of this thesis were three-fold: The first aim was to investigate the roles of various markers in the prevalence and complications of CAN and associated diseases with emphasis on diabetes mellitus. Specifically, this study investigated the role of heart rate variability (HRV) markers as well as the roles of genetic and family history risk factors. The second aim of this study was to develop mechanisms to predict CAN disease occurrence. The third aim of this current study was to develop a model for predicting diabetes mellitus (DM) and cardiovascular disease (CVD) simultaneously using common risk factors

CISDA Development Process for decision aids to support self-care decision making

The self-care management of chronic disease patients is complicated by various everyday decisions that range from routine ill-structured problems, e.g., “What to eat?” to uncertain symptoms-related decisions, e.g., “Why do I feel tired?” Such decisions can have significant consequences on a patient’s health, treatment, care, and associated medical costs. Due to the complexity involved in understanding and analysing everyday decision making, there is a lack of empirical research to guide the development of self-care decision aids. This thesis aims to address this problem by formulating and illustrating the Critical Illness Self-care Decision Aid (CISDA) process through a coherent, structured, integrated design and development process using a case study. Following a literature review, the problems in current approaches and the criteria needed for the development were derived from evidence-based frameworks such as chronic disease management, decision aids standards and complex interventions development process for future designs. Mixed methods were used including: focus groups, interviews, questionnaire, Cognitive Work Analysis and case scenarios for not only constructing an account of self-care needs and decisions but also to evaluate the development process and the decision support provided involving patients, doctors, caregivers, non-medical experts like psychologists and IT/Systems engineers. The CISDA process consists of: (i) needs assessment, (ii) theory formation, (iii) modelling, (iv) integration, (v) interface design and development, and (vi) evaluation for addressing the relevant intersection of human factors, systems engineering, and software engineering. This thesis should prove useful to not only systems engineers but also to a range of practitioners concerned about decision making, maintaining a user's cognitive perspective during specification and analysis of a complex system