Toward a document evaluation methodology: What does research tell us about the validity and reliability of evaluation methods?

Although the usefulness of evaluating documents has become generally accepted among communication professionals, the supporting research that puts evaluation practices empirically to the test is only beginning to emerge. This article presents an overview of the available research on troubleshooting evaluation methods. Four lines of research are distinguished concerning the validity of evaluation methods, sample composition, sample size, and the implementation of evaluation results during revisio

The assessment of usability of electronic shopping: A heuristic evaluation

Today there are thousands of electronic shops accessible via the Web. Some provide user-friendly features whilst others seem not to consider usability factors at all. Yet, it is critical that the electronic shopping interface is user-friendly so as to help users to obtain their desired results. This study applied heuristic evaluation to examine the usability of current electronic shopping. In particular, it focused on four UK-based supermarkets offering electronic services: including ASDA, Iceland, Sainsbury, and Tesco. The evaluation consists of two stages: a free-flow inspection and a task-based inspection. The results indicate that the most significant and common usability problems have been found to lie within the areas of ‘User Control and Freedom’ and ‘Help and Documentation’. The findings of this study are applied to develop a set of usability guidelines to support the future design of effective interfaces for electronic shopping

Reviewing and extending the five-user assumption: A grounded procedure for interaction evaluation

" © ACM, 2013. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in ACM Transactions on Computer-Human Interaction (TOCHI), {VOL 20, ISS 5, (November 2013)} http://doi.acm.org/10.1145/2506210 "The debate concerning how many participants represents a sufficient number for interaction testing is well-established and long-running, with prominent contributions arguing that five users provide a good benchmark when seeking to discover interaction problems. We argue that adoption of five users in this context is often done with little understanding of the basis for, or implications of, the decision. We present an analysis of relevant research to clarify the meaning of the five-user assumption and to examine the way in which the original research that suggested it has been applied. This includes its blind adoption and application in some studies, and complaints about its inadequacies in others. We argue that the five-user assumption is often misunderstood, not only in the field of Human-Computer Interaction, but also in fields such as medical device design, or in business and information applications. The analysis that we present allows us to define a systematic approach for monitoring the sample discovery likelihood, in formative and summative evaluations, and for gathering information in order to make critical decisions during the interaction testing, while respecting the aim of the evaluation and allotted budget. This approach – which we call the ‘Grounded Procedure’ – is introduced and its value argued.The MATCH programme (EPSRC Grants: EP/F063822/1 EP/G012393/1

Usability of Electronic Health Record-Generated Discharge Summaries: Heuristic Evaluation.

BACKGROUND: Obtaining accurate clinical information about recent acute care visits is extremely important for outpatient providers. However, documents used to communicate this information are often difficult to use. This puts patients at risk of adverse events. Elderly patients who are seen by more providers and have more care transitions are especially vulnerable. OBJECTIVE: This study aimed to (1) identify the information about elderly patients\u27 recent acute care visits needed to coordinate their care, (2) use this information to assess discharge summaries, and (3) provide recommendations to help improve the quality of electronic health record (EHR)-generated discharge summaries, thereby increasing patient safety. METHODS: A literature review, clinician interviews, and a survey of outpatient providers were used to identify and categorize data needed to coordinate care for recently discharged elderly patients. Based upon those data, 2 guidelines for creating useful discharge summaries were created. The new guidelines, along with 17 previously developed medical documentation usability heuristics, were applied to assess 4 simulated elderly patient discharge summaries. RESULTS: The initial research effort yielded a list of 29 items that should always be included in elderly patient discharge summaries and a list of 7 helpful, but not always necessary items. Evaluation of 4 deidentified elderly patient discharge summaries revealed that none of the documents contained all 36 necessary items; between 14 and 18 were missing. The documents each had several other issues, and they differed significantly in organization, layout, and formatting. CONCLUSIONS: Variations in content and structure of discharge summaries in the United States make them unnecessarily difficult to use. Standardization would benefit both patients, by lowering the risk of care transition-related adverse events, and outpatient providers, by helping reduce frustration that can contribute to burnout. In the short term, acute care providers can help improve the quality of their discharge summaries by working with EHR vendors to follow recommendations based upon this study. Meanwhile, additional human factors work should determine the most effective way to organize and present information in discharge summaries, to facilitate effective standardization

Investigating Automatic Static Analysis Results to Identify Quality Problems: an Inductive Study

Background: Automatic static analysis (ASA) tools examine source code to discover "issues", i.e. code patterns that are symptoms of bad programming practices and that can lead to defective behavior. Studies in the literature have shown that these tools find defects earlier than other verification activities, but they produce a substantial number of false positive warnings. For this reason, an alternative approach is to use the set of ASA issues to identify defect prone files and components rather than focusing on the individual issues. Aim: We conducted an exploratory study to investigate whether ASA issues can be used as early indicators of faulty files and components and, for the first time, whether they point to a decay of specific software quality attributes, such as maintainability or functionality. Our aim is to understand the critical parameters and feasibility of such an approach to feed into future research on more specific quality and defect prediction models. Method: We analyzed an industrial C# web application using the Resharper ASA tool and explored if significant correlations exist in such a data set. Results: We found promising results when predicting defect-prone files. A set of specific Resharper categories are better indicators of faulty files than common software metrics or the collection of issues of all issue categories, and these categories correlate to different software quality attributes. Conclusions: Our advice for future research is to perform analysis on file rather component level and to evaluate the generalizability of categories. We also recommend using larger datasets as we learned that data sparseness can lead to challenges in the proposed analysis proces

Translated English Online Patient Information on Finnish Public Hospital Websites : Usability and Equality in Focus

ABSTRACT Tutkielmassani arvioin englannin- ja suomenkielisten potilasohjeiden eroja ja englanninkielisten potilasohjeiden alueellisia käytettävyyseroja tasa-arvoisuuden ja käytettävyyden näkökulmista. Lisäksi vertasin operatiivisten ja informatiivisten tekstityyppien käytettävyyttä. Hypoteesina oli, että operatiivisissa tekstinäytteissä on informatiivisia tekstinäytteitä vähemmän käytettävyysongelmia niiden akuutimmasta sisällöstä johtuen. Käsitykseni englanninkielisten potilasohjeiden lukijoista perustin tilastoihin ja kirjallisuuteen. Tutustuin vähemmistökielten asemaan lainsäädännössä, ulkomaalaistaustaisen väestön määrään ja alueelliseen sijoittumiseen Suomessa sekä Suomessa käytettyjen kielten valikoimaan. Perehdyin myös instituutioissa tuotettujen sekä lääketieteen tekstien kääntämisen erityispiirteisiin. Tekstinäytteet keräsin sairaaloiden ja sairaanhoitopiirien internetsivustoilta. Analysoin aineiston laadullisesti. Analyysimenetelmiäni olivat vertaileva käännöstutkimus ja muokatut käytettävyysheuristiikat. Tutkimustuloksista kävi ilmi, että englanninkielisten potilasohjeiden käytettävyydessä on alueellisia eroja ja että käytettävyyden eri osa-alueet painottuvat eri tavoin eri alueilla. Vakavimmat käytettävyysongelmat liittyivät tekstilajikonventioihin. Ongelmia aiheuttivat virkakielisyys ja muut tekstilajille tyypilliset piirteet sekä lähtökielen tekstilajikonventioiden siirtyminen kohdekieleen. Tuloksista ilmeni myös suomen- ja englanninkielisten potilasohjeiden välisiä eroja. Tiedon saatavuus ja määrä oli kattavampi lähdekielellä. Kohteliaisuusstrategiat, rekisteri ja tyyli vaihtelivat englanninkielisissä teksteissä suomenkielisiä tekstejä enemmän siten että samassakin kohdetekstissä tyyli saattoi vaihdella muodollisen kohteliaasta puhekieliseen. Vaikka englanninkielisissä potilasohjeissa lääketieteen termejä ja fraaseja käytetiin pääosin yhdenmukaisesti, lähdekielen tekstilajikonventioiden siirtyminen kohdetekstiin aiheutti toisinaan sanan tai fraasin merkityksen laajenemista tai hämärtymistä. Kohdeteksteissä myös tekstilajille tyypilliset ei-toivotut piirteet usein vahvistuivat. Osa eroista johtunee alueiden erityispiirteistä, kuten siitä kuinka suuri osa väestöstä on muunkielisiä. Tulokset osoittivat hypoteesin oikeaksi eli operatiivisten potilasohjeiden käytettävyys oli parempi kuin informatiivisten potilasohjeiden

Defining usability heuristics for adoption and efficiency of an electronic workflow document management system

Usability heuristics have been established for different uses and applications as general guidelines for user interfaces. These can affect the implementation of industry solutions and play a significant role regarding cost reduction and process efficiency. The area of electronic workflow document management (EWDM) solutions, also known as workflow, lacks a formal definition of usability heuristics. With the advent of new technologies such as mobile devices, defining a set of usability heuristics contributes to the adoption and efficiency of an EWDM system. Workflow usability has been evaluated for various industries. Most significantly research has been done for electronic healthcare records (EHR). In other areas such as the financial sector and educational institutions there is also some literature available but not as abundant as for EHR. This was identified as a possible research limitation. The general purpose of this research was to establish and validate an overarching set of usability heuristics for EWDM in general. This was approached by conducting a literature review and a survey on 32 workflow consultants from Hyland Software, Inc. Quantitative and qualitative data was collected focusing on the study’s main research question: “what usability heuristics should be defined to ensure the adoption and efficiency of a workflow implementation? Findings based on regression testing and expert opinions have suggested a proposed set of usability heuristics. The final list consists of: adaptability to diverse platforms, user control, system feedback, intuitive interfaces, visibility on mobile devices, error management, help, and documentation

Facilitating heuristic evaluation for novice evaluators

Heuristic evaluation (HE) is one of the most widely used usability evaluation methods. The reason for its popularity is that it is a discount method, meaning that it does not require substantial time or resources, and it is simple, as evaluators can evaluate a system guided by a set of usability heuristics. Despite its simplicity, a major problem with HE is that there is a significant gap in the quality of results produced by expert and novice evaluators. This gap has made some scholars question the usefulness of the method as they claim that the evaluation results are a product of the evaluator’s experience rather than the method itself. In response, the goal of this thesis is to bridge the gap between expert and novice evaluators. Based on interviews with 15 usability experts, which focused on their experience with the method, the difficulties they faced when they were novices, and how they overcame such difficulties, it presents a comprehensive protocol called Coherent Heuristic Evaluation (CoHE). This step-by-step protocol guides novice evaluators from the moment they decide to conduct an evaluation until the submission of their evaluation report. This protocol was verified by conducting an experiment to observe the difference between novices using the CoHE protocol and novices using Nielsen’s 10 usability heuristics without the guidance. The experiment involved 20 novices performing two sessions; the first was an understanding session where the novices read and understood the heuristics and the second was an inspecting session where they inspected a system. The findings show that, while evaluators take more time to read and evaluate a system using CoHE, they tend to identify more problems. The experiment also demonstrated that CoHE can improve the thoroughness, effectiveness, and f-measure of evaluation. However, the validity of CoHE was comparable to that of HE