Evaluation methods used on health information systems (HISs) in Iran and the effects of HISs on Iranian healthcare: a systematic review.

Objectives: The most important goal of a health information system (HIS) is improvement of quality, effectiveness and efficiency of health services. To achieve this goal, health care systems should be evaluated continuously. The aim of this paper was to study the impacts of HISs in Iran and the methods used for their evaluation. Methods: We systematically searched all English and Persian papers evaluating health information systems in Iran that were indexed in SID, Magiran, Iran medex, PubMed and Embase databases until June 2013. A data collection form was designed to extract required data such as types of systems evaluated, evaluation methods and tools. Results: In this study, 53 out of 1103 retrieved articles were selected as relevant and reviewed by the authors. This study indicated that 28 studies used questionnaires to evaluate the system and in 27 studies the study instruments were distributed within a research population. In 26 papers the researchers collected the information by means of interviews, observations, heuristic evaluation and the review of documents and records. The main effects of the evaluated systems in health care settings were improving quality of services, reducing time, increasing accessibility to information, reducing costs and decreasing medical errors. Conclusion: Evaluation of health information systems is central to their development and enhancement, and to understanding their effect on health and health services. Despite numerous evaluation methods available, the reviewed studies used a limited number of methods to evaluate HIS. Additionally, the studies mainly discussed the positive effects of HIS on health care services

CPOE System Design Aspects and Their Qualitative Effect on Usability

Although many studies have discussed the benefits of Computerized Provider Order Entry (CPOE) systems, their configuration can have a great impact on clinicians’ adoption of these systems. Poorly designed CPOE systems can lead to usability problems, users’ dissatisfaction and may disrupt normal flow of clinical activities. This paper reports on a literature review focused on the identification of CPOE medication systems’ design aspects that impact CPOE systems’ usability and create opportunities for medication errors. Our review is based on a systematic literature search in PubMed, EMBASE and Ovid MEDLINE for relevant publications from 1986-2006. We categorized the design aspects extracted from relevant publications into six different groups: 1) timing of alerts, 2) log in/out procedures, 3) pick lists and drop down menus, 4) clues and guidelines, 5) documentation and data entry options, and 6) screen display and layout. Our review shows that the manner in which a CPOE system is configured can have a high impact on ease of system use, task behavior of clinicians in ordering drugs, and medication errors. Characterization of consequences associated with certain CPOE design aspects provides insight into how CPOE system designs can be improved to enhance physicians’ adoption of these systems and their success. Recommendations are provided to enable CPOE system designers to create CPOE systems that are not only more user friendly and efficient but safer

Usability Evaluation of a Computerized Physician Order Entry for Medication Ordering

Despite CPOE (Computerized Physician Order Entry) systems’ potential to enhance patient safety by reducing medication errors, recent studies have cast some doubts on their role in error reduction. CPOE systems with poorly designed interfaces have proven to cause users dissatisfaction and to introduce new kind of errors in the ordering process, suggesting a threat instead of an enhancement of patient safety. The main objective of this study is to identify usability problems related to a CPOE medication system’s design and determining their severities. Two experts completed a cognitive walkthrough (CW) of an ordering task based on a clinical scenario for ordering the consolidation phase of chemotherapy for a leukemic patient. Fifty five usability problems were found and classified into eleven categories. CW identified cosmetic to catastrophic problems leading to inefficient use of the CPOE system and potentially resulting in users’ confusion, longer ordering duration, and medication errors. The complexity of the CPOE design, its rigidness and lack of user guidance suggests the necessity to redesign the current user interface in order to match clinicians’ ordering behaviors and to fully support them in the medication ordering process

Electronic Prescribing Usability: Reduction of Mental Workload and Prescribing Errors Among Community Physicians

Background: Medical errors are common in hospitals, and research is always needed to find ways of reducing these. This study attempts to address three gaps in this field. First, the factors leading to the reduction of mental workload and its relationship with the reduction of prescribing errors by improving electronic prescribing (e-prescribing) usability have not been empirically examined before. Second, the past research in the field of e-prescribing usability lacks robust theoretical models. Third, there are no existing studies to examine the direct influences of user interface consistency and error prevention with the reduction of mental workload and prescribing errors. Materials and Methods: A quantitative survey method was used to collect data from 188 community physicians. The partial least squares path modeling technique was applied to analyze the data. Results: Prescribing errors were reduced by improving the information quality, user interface consistency, system ease of use, and mental workload reduction. Mental workload is reduced by ease of use, error prevention, and consistency. No significant relationships between prescribing error reduction with error prevention and also between information quality with mental workload reduction were found. Conclusions: The designers of e-prescribing should improve the error prevention and consistency of the system and make it easy to use if they wish for the system to reduce users’ mental workload. They should also improve the system information quality, ease of use, and consistency if they claim that their system reduces physicians’ prescribing errors. The system should also reduce users’ mental workload to meet this objective

Usabiility evaluation of healthcare information systems. Comparison of methods and classification of usability problems

In ziekenhuizen wordt steeds meer gebruik gemaakt van informatiesystemen, maar de usability laat vaak te wensen over. Reza Khajouei testte twee methodes om de usability te beoordelen. Hij constateert dat beide methoden ongeveer evenveel, maar verschillende problemen aan het licht brengen. Uit een enquête onder artsen en verpleegkundigen blijkt dat er veel problemen zijn met het gebruik van de informatiesystemen, maar dat men uiteindelijk toch positief is

Assessment of the Performance of the Laboratory Information System (LIS) Based on the Standards of the American National Standards Institute (ANSI)

Introduction: LIS is a system developed for the purpose of collecting, processing, storing, and retrieving the data created in laboratory context. Accordingly, this research intends to evaluate the performance of the LIS based on the standards of the ‎American National Standards Institute (ANSI) in the educational and private hospitals situated in the city of Isfahan. Method: This study is applied and descriptive-analytical in nature. The data were collected using a self-designed checklist prepared on the basis of standards of the ANSI including LIS-8A and LIS2-A, the validity and reliability of which were assessed and confirmed by the experts on laboratory and computer sciences as well as the department’s professors. The data were analyzed using SPSS 20 software using some statistical tests including T-test and ANOVA test so as to examine the informational content of the LISs based on three criteria, namely, data input, data storage, and processing and data output using the foregoing standard. In addition, LEVEN test was used for checking the equality of the variances. Results: Based on the findings of the study, the mean score for the three criteria (data input, data storage and processing, and data output) examined, referring to the standards, was found to be 27.53. As a result, the private and educational hospitals under study did not show any difference in this regard. Furthermore, among the systems in question and in terms of the level of meeting three standards, Kowsar system (%44.5) enjoyed the highest rank compared to other software systems. Conclusion: Given the results, it was found that the standards under study for the LISs have not been given due attention. Hence, it is recommended that the proposed standards be examined and studied by organizations so that a user-friendly and responsive system can be developed based on the results of the needs-analysisdone from the beneficiary organizations

Online Focus Groups for the Development of a Usability Evaluation Tool: Lessons Learned

Introduction: Focus group discussions are a well-established method for acquiring insights from experts in different fields. This method requires special amendments when it is used for the development of different tools in the healthcare domain. The objective of this paper is to present the lessons learned from online focus group sessions held for the development of a heuristic usability evaluation tool for mobile health applications. Material and Methods: Two online focus group sessions were conducted with the participation of ten medical informatics experts to develop the tool. The sessions were recorded using screen recording software. The comments provided by the experts were categorized, and the lessons learned from these sessions were identified and reported. Results: The experiences achieved from the online focus group sessions were categorized into the following ten lessons. 1. Engage the participants fully in online session discussions 2. Use an appropriate and interesting format. 3. Select an appropriate number of people for online sessions 4. Invite people having the closest expertise related to the research topic 5. Employ a technical support technician in addition to the coordinator. 6. Prevent the emergence of a new topic in sessions. 7. Arrange the required hardware and software facilities before the session. 8. Prepare the content in an appropriate language. 9. Use online tools to schedule sessions.  10. Use screen-recording software. Conclusion: This paper reports the lessons learned from holding online focus group sessions in the process of developing a heuristic usability evaluation tool for mobile health applications. Although these lessons were learned in a study focusing on the development of a usability tool, they can also be used to improve the results of focus group sessions in other fields of medical informatics

Proposing an initial model for the heuristic evaluation of mHealth applications

Background: Health-related applications are concerned with human life. These applications should be developed and evaluated to minimize any potential risk while applied. One of the most common usability evaluation methods is known as heuristic evaluation. Thus, the present study aimed to propose an initial model for the heuristic evaluation of health-related applications. Methods: To propose this model, the existing methodologies were modified. In the present study, the three primary stages of the methodology were reported. In the first and second stages, having identified the relevant keywords, a search was done in the databases. After the initial screening and extraction of heuristics and important criteria in the first two stages, the research findings and data abstraction were done qualitatively in the third stage. Results: A total number of twenty-one heuristics in the proposed model were mapped into two categories, traditional and non-traditional. Ten heuristics were placed in the former and eleven in the latter. Flexibility and efficiency of use, Aesthetic and minimalist design heuristics in the first category and User engagement and Privacy and security heuristics in the second category had the largest number of items. Technical features and support heuristic were also added to the second stage. Conclusion: This initial model can be used by healthcare providers, engineers, developers, researchers and health policymakers. Using a pre-defined model or framework to evaluate health-related applications saves time and money

User Interface Problems of a Nationwide Inpatient Information System: A Heuristic Evaluation

Introduction: While studies have shown that usability evaluation could uncover many design problems of health information systems, the usability of health information systems in developing countries using their native language is poorly studied. The objective of this study was to evaluate the usability of a nationwide inpatient information system used in many academic hospitals in Iran. Material and Methods: Three trained usability evaluators independently evaluated the system using Nielsen’s 10 usability heuristics. The evaluators combined identified problems in a single list and independently rated the severity of the problems. We statistically compared the number and severity of problems identified by HIS experienced and non-experienced evaluators. Results: A total of 158 usability problems were identified. After removing duplications 99 unique problems were left. The highest mismatch with usability principles was related to “Consistency and standards” heuristic (25%) and the lowest related to “Flexibility and efficiency of use” (4%). The average severity of problems ranged from 2.4 (Major problem) to 3.3 (Catastrophe problem). The experienced evaluator with HIS identified significantly more problems and gave higher severities to problems (p<0.02). Discussion: Heuristic Evaluation identified a high number of usability problems in a widely used inpatient information system in many academic hospitals. These problems, if remain unsolved, may waste users’ and patients’ time, increase errors and finally threaten patient’s safety. Many of them can be fixed with simple redesign solutions such as using clear labels and better layouts. This study suggests conducting further studies to confirm the findings concerning effect of evaluator experience on the results of Heuristic Evaluation