Designing Clinical Data Presentation Using Cognitive Task Analysis Methods

Despite the many decades of research on effective use of clinical systems in medicine, the adoption of health information technology to improve patient care continues to be slow especially in ambulatory settings. This applies to dentistry as well, a primary care discipline with approximately 137,000 practicing dentists in the United States. One critical reason is the poor usability of clinical systems, which makes it difficult for providers to navigate through the system and obtain an integrated view of patient data during patient care. Cognitive science methods have shown significant promise to meaningfully inform and formulate the design, development and assessment of clinical information systems. Most of these methods were applied to evaluate the design of systems after they have been developed. Very few studies, on the other hand, have used cognitive engineering methods to inform the design process for a system itself. It is this gap in knowledge – how cognitive engineering methods can be optimally applied to inform the system design process – that this research seeks to address through this project proposal. This project examined the cognitive processes and information management strategies used by dentists during a typical patient exam and used the results to inform the design of an electronic dental record interface. The resulting 'proof of concept' was evaluated to determine the effectiveness and efficiency of such a cognitively engineered and application flow design. The results of this study contribute to designing clinical systems that provide clinicians with better cognitive support during patient care. Such a system will contribute to enhancing the quality and safety of patient care, and potentially to reducing healthcare costs

Online Reading Comprehension Among Seventh Grade Students with High Incidence Disabilities in Inclusive Settings: A Mixed Methods Study

Because research exploring how students with disabilities read and comprehend on the Internet is scarce, a mixed methods study was implemented to determine if Internet Reciprocal Teaching (IRT) is an effective intervention for improving online reading comprehension among seventh grade students with high-incidence disabilities in inclusive settings. Differences between students with disabilities and their non-disabled peers were also explored. The intervention included a twenty week (40 lessons) instructional program delivered in three phases in seventh grade English/Language Arts classes from three middle schools in eastern region of the United States. Pre and post intervention data was collected on the Online Reading Comprehension Assessment (ORCA-Iditarod) and the Survey of Online Reading. Further, and a sample of students was randomly selected for further post-intervention qualitative analysis. Qualitative and quantitative data were then triangulated to examine convergent and divergent findings of online reading comprehension. Results indicate that online reading comprehension, as measured by the ORCA-Iditarod increased for students in the treatment group, but no apparent differences appeared between general education students and students with disabilities. Students in the treatment group demonstrated increased self-efficacy of reading online and locating answers. Qualitative findings further supported improvements in online reading comprehension noting more frequent use of effective search strategies among students in the treatment group, more effective strategy use for determining the reliability of Web sites, and improved communicating strategies using email. While the ORCA did not reveal significant differences between students with disabilities and their nondisabled peers, Survey data indicated that students with disabilities are using and receiving more instruction on the Internet at school, and some qualitative results revealed more attention to written mechanics and spell check tools than general education students. Implications for practice and recommendations for future research are provided

Understanding Children’s Help-Seeking Behaviors: Effects of Domain Knowledge

This dissertation explores children’s help-seeking behaviors and use of help features when they formulate search queries and evaluate search results in IR systems. This study was conducted with 30 children who were 8 to 10 years old. The study was designed to answer three research questions with two parts in each: 1(a) What are the types of help-seeking situations experienced by children (8-10 years old) when they formulate search queries in a search engine and a kid-friendly web portal?, 1(b) What are the types of help-seeking situations experienced by children (8-10 years old) when they evaluate search results in a search engine and a kid-friendly web portal?, 2(a) What types of help features do children (8-10 years old) use and desire when they formulate search queries in a search engine and a kid-friendly web portal?, 2(b) What types of help features do children (8-10 years old) use and desire when they evaluate search results in a search engine and a kid-friendly web portal?, 3(a) How does children’s (8-10 years old) domain knowledge affect their help seeking and use of help features when they formulate search queries in a search engine and a kid-friendly web portal?, 3(b) How does children’s (8-10 years old) domain knowledge affect their help seeking and use of help features when they evaluate search results in a search engine and a kid-friendly web portal? This study used multiple data collection methods including performance-based domain knowledge quizzes as direct measurement, domain knowledge self-assessments as indirect measurement, pre-questionnaires, transaction logs, think-aloud protocols, observations, and post-interviews. Open coding analysis was used to examine children’s help-seeking situations. Children’s cognitive, physical, and emotional types of help-seeking situations when using Google and Kids.gov were identified. To explore help features children use and desire when they formulate search queries and evaluate results in Google and Kids.gov, open coding analysis was conducted. Additional descriptive statistics summarized the frequency of help features children used when they formulated search queries and evaluated results in Google and Kids.gov. Finally, this study investigated the effect of children’s domain knowledge on their help seeking and use of help features in using Google and Kids.gov based on linear regression. The level of children’s self-assessed domain knowledge affects occurrences of their help-seeking situations when they formulated search queries in Google. Similarly, children’s domain knowledge quiz scores showed a statistically significant effect on occurrences of their help-seeking situations when they formulated keywords in Google. In the stage of result evaluations, the level of children’s self-assessed domain knowledge influenced their use of help features in Kids.gov. Furthermore, scores of children’s domain knowledge quiz affected their use of help features when they evaluated search results in Kids.gov. Theoretical and practical implications for reducing children’s cognitive, physical, and emotional help-seeking situations when they formulate search queries and evaluate search results in IR systems were discussed based on the results

Developing an understanding of the nature of accessibility and usability problems blind students face in web-enhanced instruction environments

The central premise of this research is that blind and visually impaired (BVI) people cannot use the Internet effectively due to accessibility and usability problems. Use of the Internet is indispensable in today's education system that relies on Web-enhanced instruction (WEI). Therefore, BVI students cannot participate effectively in WEI. Extant literature recognizes that non-visual Web interaction is inherently challenging. However, it does not explain where, how and why BVI students face accessibility and usability problems in performing academic tasks in WEI. This knowledge is necessary to adequately inform the development of interventions that improve the functional and academic outcomes of BVI students in WEI. The purpose of this doctoral research is to understand the nature of accessibility and usability problems BVI students face in WEI environments. It adopts a novel user-centered, task-oriented, cognitive approach to develop an in-depth, contextually-situated, observational and experiential knowledge of these problems. The context of WEI experience under investigation is an online exam over a typical course management system. Research design is a qualitative field study that involves a multimethod evaluation of the WEI environment. The core component of this multimethod evaluation is BVI students' assessment of the WEI environment. This is triangulated through assessments made by WCAG (Web Content Accessibility Guidelines) and Web developers. The BVI student assessment employs an integrated problem solving model, in combination with verbal protocol analysis, to identify and understand where, how and why BVI students face a problem in completing the exam. The WCAG assessment employs automated accessibility testing and WCAG textual analysis to identify interface objects that violate accessibility standards and characterizes a problem. The Web developer assessment involves open-ended interviews to identify the source of a problem. Results show that the WEI environment consisted of innumerable interface objects that violated WCAG's standards on Web accessibility and usability. BVI participants faced many accessibility and usability problems that posed significant challenges completing the online exam. These problems fall into six major problem types as described below: 1. Confusion while navigating across WEI environment due to its frame-based page structure without unique frame names; 2. Susceptibility to submitting incomplete work when a new question page does not provide location and contextual information; 3. Difficulty understanding how to submit work when the selection controls for multiple option questions lack a consistent keyboard navigation procedure; 4. Inability to negotiate security information pop-up when the WEI environment uses an alert dialogue box; 5. Ambiguity in essay-type question page that lack meaningful labels for interface objects, including text area and text formatting toolbar; 6. Vulnerability of losing work when Backspace behaves as browser's Back button inside text area. This doctoral research contributes in three ways. It fills the knowledge gap about the nature of problems BVI students face in Web interactions for academic tasks. This kind of knowledge is necessary to determine accessibility and usability requirements for WEI. Another contribution is a set of mental model representations that explicate the thought processes of BVI students. Such representations are useful in developing user instruction and design of more accessible and usable Web sites. A third contribution is a user-centered, task-based, cognitive and multi-method approach to evaluate Web accessibility and usability

Navigation and wayfinding in learning spaces in 3D virtual worlds

There is a lack of published research on the design guidelines of learning spaces in virtual worlds. Therefore, when institutions aspire to create learning spaces in Second Life, there are few studies or guidelines to inform them except for individual case studies. The Design of Learning Spaces in 3D Virtual Environments (DELVE) project, funded by the Joint Information Systems Committee in the UK, was one of the first initiatives that identified through empirical investigations the usability problems associated with learning spaces in virtual worlds and the potential impact on student experience. The findings of the DELVE project revealed that applying architectural principles of real-world designs to virtual worlds may not be sufficient. In fact, design principles from urban planning, Human–Computer Interaction (HCI), web usability, geography, and psychology influence the design of learning spaces in virtual worlds. In DELVE, the researchers derived several usability guidelines: form should follow function, that is, that the shape of a building or object should be primarily based upon its intended function or purpose; use real-world metaphors such as mailboxes for students to leave messages, or search pods similar to real-world information kiosks; consider realism for familiarity and comfort; design for storytelling; or design to orient the user at the landing point, etc. However, the investigations in DELVE identified that the key usability problems experienced by users in 3D learning spaces are related to navigation and wayfinding. In this chapter, we report on the Navigation and Wayfinding (NAVY) project which builds on the findings of the DELVE project. As the most commonly used virtual world for education, Second Life was the logical choice for conducting the NAVY project research. Based upon empirical investigations of a number of islands in Second Life (an island is a space which is analogous to a website in a 2D environment) involving user-based studies, heuristic evaluations, and iterative reviews of the heuristics by usability experts, we have derived over 200 guidelines for the design of learning spaces in virtual worlds.

Thinking about thinking aloud : an investigation of think-aloud methods in usability testing

In website design and engineering, the term “usability” describes how easy a website or interface is to use. As the Internet continues to grow exponentially, with millions of websites vying for users’ attention, usability has become a critical factor determining whether a website will survive or fail. If websites are not sufficiently usable, users will simply abandon them in favour of alternatives that better cater to their needs. It is therefore crucial that designers employ effective evaluation methods in order to assess usability and improve user interface design. One of the most widely used methods of evaluating the usability of websites is the Thinking Aloud protocol, wherein users are encouraged to verbalise their experiences, thoughts, actions, and feelings whilst interacting with the design. This provides direct insight into the cognitive processes employed by users—knowledge which can then inform strategies to improve usability. However, despite the common usage of Thinking Aloud protocol in the field, the specific think-aloud procedures employed vary widely among usability professionals. The aim of this thesis is to investigate the utility and validity of the different variations of think-aloud usability testing methods. To this end, three empirical studies were conducted, using library websites, to compare the practical benefits of the various methods. The studies measured five points of comparison: overall task performance, the experiences of the test participants, the quantity and quality of usability problems discovered, the costs of employing the method in question, and the relationship between sample size and the number of problems detected. Study One examined three classic think-aloud methods: concurrent think-aloud, retrospective think-aloud, and a hybrid method. The results revealed that the concurrent method outperformed both the retrospective method and the hybrid method in facilitating successful usability testing. It detected higher numbers of usability problems than the retrospective method, and produced output comparable to that of the hybrid method. The method received average to positive ratings from its users, and no reactivity (a potential issue wherein the act of verbalising the cognitive process alters that process) was observed. In addition, this method required much less time on the evaluator’s part than did the other two methods, which involved double the testing and analysis time. Lastly, in terms of the relationship between the sample size and the number of problems discovered, the concurrent and the hybrid methods showed similar patterns, and both outperformed the retrospective method in this regard. Study Two compared the performance of the classic concurrent think-aloud method with two variations on this method in which the evaluator plays a more active role—namely, the active intervention method and the speech-communication method. The results showed that these three methods enabled the identification of a similar number of usability problems and types, and showed similar patterns with regard to the relationship between the sample size and the number of problems discovered. However, the active intervention method was found to cause some reactivity, modifying participants’ interactions with the interface, and negatively affecting their feelings towards the evaluator. The active intervention method also required much greater investment than did the other two methods, both in terms of evaluators' time, and, it was estimated, in financial terms. Study Three compared the classic concurrent think-aloud method with the co-participation method, wherein a pair of participants work together to perform their tasks, and verbalise their processes as they interact with the interface and with one another. This study found no difference between the methods in terms of task performance. However, the co-participation method was evaluated more positively by users in comparison with the classic method. It led to the detection of more minor usability problems, and performed better in terms of the relationship between the sample size and the number of problems detected. The co-participation method was, however, found to require a greater investment of time on the part of the evaluato

A Culturally Aware Approach to Learning System Interface Design

This mixed methods research explored interface design strategies for users from different cultures and localized settings. Guided by the cultural-historical development theory and HCI research, four critical factors—navigation design, information organization, layout design, and visuals—were investigated in designing culturally relevant interfaces for Americans and Taiwanese. American and Taiwanese groups—both contained two sub-groups of 30 participants—were recruited for the quantitative phase. Each participant was exposed to only one interface with content composed in their native language. However, one sub-group in each ethnic group was exposed to a culturally relevant interface and another was exposed to an alien interface. MANOVA on overall performance in both American and Taiwanese groups were significant. Americans performed better using the American interface (Wilks’s Λ=.85, F= 5.15, p< .01). They had significantly shorter performance time in the American (M=775) than the Taiwanese (M=1003) interface (F=6.29, p<.05), but differences on performance accuracy were not significant (F=2.74, p=.103). Taiwanese performed better using the Taiwanese interface (Wilks’s Λ=.67, F=14.06, p< .01). They had shorter performance time in the Taiwanese (M=743) than the American (M=1353) interface (F=6.29, p<.05), and they also had higher performance accuracy on the Taiwanese (M=11.7) than the American (M=10.0) interface (F=7.94, p<.01). In addition, t-test on overall preference in both American and Taiwanese groups were significant. Americans preferred the American (M=58.5) over the Taiwanese (M=53.0) interface (t=2.11, p< .05). And Taiwanese preferred the Taiwanese (M=58.7) over the American (M=46.9) interface (t=3.48, p<.01). Qualitative interviews of six American and six Taiwanese participants revealed three themes: First, when searching, Taiwanese were explorative and relied on hierarchical relationships; while Americans relied on prior experiences and analytical categorizations. Second, both groups have higher affiliation with design features matching their preferences. Finally, matching design features with users’ expectations and needs promotes positive perceptions and enhances interface usability. Both quantitative and qualitative Results imply that user interface designers should consider cultural perspectives when designing interfaces for online learning systems. Further studies might consider the relative impacts of the navigation, information structure, layout, and visual design on a broad range of user differences might have on learning

Linguistic Representation of Problem Solving Processes in Unaided Object Assembly

This thesis investigates the linguistic representation of problem solving processes in data recorded during unaided object assembly. It combines traditional approaches of analyzing verbal protocols with the recent approach of Cognitive Discourse Analysis

Development and Usability Evaluation of an mHealth Application for Symptom Self-Management in Underserved Persons Living with HIV

Effective symptom management is essential to decrease symptom severity and improve health-related quality of life for persons living with HIV (PLWH). A mobile health (mHealth) application (app) has the potential to be an effective delivery mode of an existing paper-based symptom management manual with self-management strategies for underserved PLWH. The quality of the mHealth app requires a thorough understanding of the needs of the intended end-users and ensuring the app’s usability. The purpose of this study was to translate paper-based health information into an mHealth app for symptom self-management in underserved PLWH, entitled mVIP (mobile Video Information Provider), and assess its usability. To achieve this goal, usability was evaluated rigorously throughout the development process of mVIP. Based on a stratified view of health information technology (IT) usability evaluation framework, usability evaluation was sequentially conducted with the following three levels: 1) user-task, 2) user-task-system, and 3) user-task-system-environment. At level 1 (user-task), we applied a user-centered design method to guide the information architecture of mVIP. Using a reverse in-person card sorting technique, symptoms and self-management strategies from a paper-based HIV/AIDS symptom management manual were ranked. The rank order of the 13 symptoms and 151 self-management strategies determined the order of appearance to end-users of the mVIP app, with higher-ranked symptoms and strategies appearing first. Based on the findings, we developed a prototype of mVIP as following: 1) once users log in, they are guided by an avatar through a series of 13 symptom questions ascertaining the nature and severity of their symptoms, and 2) the avatar recommends three self-management strategies for each symptom reported. At level 2 (user-task-system), we conducted a usability evaluation of the mVIP prototype in a laboratory setting through end-user usability testing and heuristic evaluation. In end-user usability testing, we used an eye-tracking and retrospective think-aloud method to examine task performance by 20 PLWH. For the heuristic evaluation, five usability experts in informatics assessed the user interface. In the two usability evaluations conducted in a laboratory setting, we found strong user acceptance of the mVIP prototype while identifying a number of usability issues with this prototype. Based on the recommendations from the end-users and heuristic evaluators, we iteratively refined the app’s content, functionality, and interface. We then inserted videos of the finalized symptom self-management strategies into the refined mVIP prototype. At level 3 (user-task-system-environment), the usability of the refined mVIP prototype was evaluated in a real-world setting. Through 10 in-depth interviews and four focus groups conducted at the conclusion of a three-month randomized controlled trial, we explored in-depth understandings of users’ experiences, perceptions, and satisfaction of mVIP use. Findings from the study showed that first, mVIP is useful for HIV-related symptom self-management and has the potential for being used as a communication tool with healthcare providers; and second, mVIP is easy to use to monitor symptom experience over time. At the same time, participants suggested mVIP be more sensitively tailored based on years from initial diagnosis of HIV, an individuals’ age, and conditions. The overall user satisfaction with the mVIP prototype was high, which reflects strong user acceptance of mVIP. Integral to the findings from the three-level usability evaluation, we assessed the quality of the mVIP prototype in use and found the prototype was highly accepted by PLWH with high user satisfaction. This study will add to the body of literature on translation of evidence-based health information into an mHealth app and its usability assessment, which highlights the importance of the use of mobile technology for PLWH, specifically racial and ethnic minorities and those from low-socioeconomic groups who have limited health literacy and low level of education