Task-oriented evaluation of electronic medical records systems: development and validation of a questionnaire for physicians

BACKGROUND: Evaluation is a challenging but necessary part of the development cycle of clinical information systems like the electronic medical records (EMR) system. It is believed that such evaluations should include multiple perspectives, be comparative and employ both qualitative and quantitative methods. Self-administered questionnaires are frequently used as a quantitative evaluation method in medical informatics, but very few validated questionnaires address clinical use of EMR systems. METHODS: We have developed a task-oriented questionnaire for evaluating EMR systems from the clinician's perspective. The key feature of the questionnaire is a list of 24 general clinical tasks. It is applicable to physicians of most specialties and covers essential parts of their information-oriented work. The task list appears in two separate sections, about EMR use and task performance using the EMR, respectively. By combining these sections, the evaluator may estimate the potential impact of the EMR system on health care delivery. The results may also be compared across time, site or vendor. This paper describes the development, performance and validation of the questionnaire. Its performance is shown in two demonstration studies (n = 219 and 80). Its content is validated in an interview study (n = 10), and its reliability is investigated in a test-retest study (n = 37) and a scaling study (n = 31). RESULTS: In the interviews, the physicians found the general clinical tasks in the questionnaire relevant and comprehensible. The tasks were interpreted concordant to their definitions. However, the physicians found questions about tasks not explicitly or only partially supported by the EMR systems difficult to answer. The two demonstration studies provided unambiguous results and low percentages of missing responses. In addition, criterion validity was demonstrated for a majority of task-oriented questions. Their test-retest reliability was generally high, and the non-standard scale was found symmetric and ordinal. CONCLUSION: This questionnaire is relevant for clinical work and EMR systems, provides reliable and interpretable results, and may be used as part of any evaluation effort involving the clinician's perspective of an EMR system

Evaluation of electronic medical records - A clinical task perspective

Evaluation is a challenging but necessary part of the development cycle of clinical information systems like the complex electronic medical records (EMR) systems. It is believed that evaluations of EMR systems should include multiple perspectives, be comparative and employ both qualitative and quantitative methods. Self-administered questionnaires are frequently used as a quantitative evaluation method in medical informatics, but very few validated questionnaires address clinical use of EMR systems, and comparative investigations are scarce. A task-oriented questionnaire has been developed for evaluating EMR systems from the physician’s perspective. The key feature of the questionnaire is a list of 24 general clinical tasks. The list of tasks is applicable to physicians of most specialties and covers essential parts of their information-oriented work. The list appears as in two separate sections, about EMR use and task performance using the EMR, respectively. Using the questionnaire, the evaluator may quickly estimate the potential impact of the EMR system on health care delivery. Problematic areas may be found by identifying clinical tasks for which the EMR system either is not used, or for which performing the task is more difficult when using the system. These results may be compared across time, site or vendor. The development, application and validation of the questionnaire is described in this thesis. Its performance is demonstrated in a national and a local study. In addition to underscoring the performance of the questionnaire, the demonstration studies had interesting results of their own. The national study showed that a considerable proportion of the functionality offered by the EMR systems is not used by the physicians. The local study showed that scanning and eliminating the paper-based medical record in middle-sized hospital is feasible. All physicians used the EMR system more much frequently, and while a considerable proportion of the internists found important tasks more difficult, most physicians found their EMR-supported tasks easier to perform. However, the medical secretaries in this hospital were considerably more satisfied with the system, and overall seemed to benefit more from this change in the work environment than both the physicians and the nurses. The questionnaire presented here may be used as part of any evaluation effort involving the clinician’s perspective of an EMR system

Evaluation of electronic medical records - A clinical task perspective

Evaluation is a challenging but necessary part of the development cycle of clinical information systems like the complex electronic medical records (EMR) systems. It is believed that evaluations of EMR systems should include multiple perspectives, be comparative and employ both qualitative and quantitative methods. Self-administered questionnaires are frequently used as a quantitative evaluation method in medical informatics, but very few validated questionnaires address clinical use of EMR systems, and comparative investigations are scarce. A task-oriented questionnaire has been developed for evaluating EMR systems from the physician’s perspective. The key feature of the questionnaire is a list of 24 general clinical tasks. The list of tasks is applicable to physicians of most specialties and covers essential parts of their information-oriented work. The list appears as in two separate sections, about EMR use and task performance using the EMR, respectively. Using the questionnaire, the evaluator may quickly estimate the potential impact of the EMR system on health care delivery. Problematic areas may be found by identifying clinical tasks for which the EMR system either is not used, or for which performing the task is more difficult when using the system. These results may be compared across time, site or vendor. The development, application and validation of the questionnaire is described in this thesis. Its performance is demonstrated in a national and a local study. In addition to underscoring the performance of the questionnaire, the demonstration studies had interesting results of their own. The national study showed that a considerable proportion of the functionality offered by the EMR systems is not used by the physicians. The local study showed that scanning and eliminating the paper-based medical record in middle-sized hospital is feasible. All physicians used the EMR system more much frequently, and while a considerable proportion of the internists found important tasks more difficult, most physicians found their EMR-supported tasks easier to perform. However, the medical secretaries in this hospital were considerably more satisfied with the system, and overall seemed to benefit more from this change in the work environment than both the physicians and the nurses. The questionnaire presented here may be used as part of any evaluation effort involving the clinician’s perspective of an EMR system

Effects of Scanning and Eliminating Paper-based Medical Records on Hospital Physicians' Clinical Work Practice

Objective: It is not automatically given that the paper-based medical record can be eliminated after the introduction of an electronic medical record (EMR) in a hospital. Many keep and update the paper-based counterpart, and this limits the use of the EMR system. The authors have evaluated the physicians' clinical work practices and attitudes toward a system in a hospital that has eliminated the paper-based counterpart using scanning technology. Design: Combined open-ended interviews (8 physicians) and cross-sectional survey (70 physicians) were conducted and compared with reference data from a previous national survey (69 physicians from six hospitals). The hospitals in the reference group were using the same EMR system without the scanning module. Measurements: The questionnaire (English translation available as an online data supplement at <www.jamia.org>) covered frequency of use of the EMR system for 19 defined tasks, ease of performing them, and user satisfaction. The interviews were open-ended. Results: The physicians routinely used the system for nine of 11 tasks regarding retrieval of patient data, which the majority of the physicians found more easily performed than before. However, 22% to 25% of the physicians found retrieval of patient data more difficult, particularly among internists (33%). Overall, the physicians were equally satisfied with the part of the system handling the regular electronic data as that of the physicians in the reference group. They were, however, much less satisfied with the use of scanned document images than that of regular electronic data, using the former less frequently than the latter. Conclusion: Scanning and elimination of the paper-based medical record is feasible, but the scanned document images should be considered an intermediate stage toward fully electronic medical records. To our knowledge, this is the first assessment from a hospital in the process of completing such a scanning project

Use of and attitudes to a hospital information system by medical secretaries, nurses and physicians deprived of the paper-based medical record: a case report

Abstract Background Most hospitals keep and update their paper-based medical records after introducing an electronic medical record or a hospital information system (HIS). This case report describes a HIS in a hospital where the paper-based medical records are scanned and eliminated. To evaluate the HIS comprehensively, the perspectives of medical secretaries and nurses are described as well as that of physicians. Methods We have used questionnaires and interviews to assess and compare frequency of use of the HIS for essential tasks, task performance and user satisfaction among medical secretaries, nurses and physicians. Results The medical secretaries use the HIS much more than the nurses and the physicians, and they consider that the electronic HIS greatly has simplified their work. The work of nurses and physicians has also become simplified, but they find less satisfaction with the system, particularly with the use of scanned document images. Conclusions Although the basis for reference is limited, the results support the assertion that replacing the paper-based medical record primarily benefits the medical secretaries, and to a lesser degree the nurses and the physicians. The varying results in the different employee groups emphasize the need for a multidisciplinary approach when evaluating a HIS.</p