Modelo de Objetos do openEHR: uma Revisão Sistemática da Literatura e sua relação com métricas de software

Dos principais padrões em sistemas de Registro Eletrônico de Saúde (RES), destaca-se a abordagem da Fundação openEHR. Este trabalho apresenta uma Revisão Sistemática da Literatura dos estudos que utilizam esta abordagem com ênfase na utilização do seu Modelo de Objetos. A pesquisa foi realizada nas bases de dados internacionais com base em quatro questões de pesquisa e critérios de inclusão e exclusão definidos. Entre os resultados obtidos, foi possível observar que o continente europeu é o maior centro dos estudos relacionados com a abordagem openEHR, com exceção da Austrália na Oceania. Pode-se concluir que uma versão estável da especificação openEHR contribuiu para o aumento de estudos a partir de 2008. Em relação às métricas de software aplicadas ao modelo do openEHR, até a realização deste trabalho, não se observaram estudos dessa natureza. Além disso, esta revisão possibilitou relacionar as ferramentas para coleta de métricas disponíveis na literatura

Clinical information modeling processes for semantic interoperability of electronic health records: systematic review and inductive analysis

This is a pre-copyedited, author-produced PDF of an article accepted for publication in Journal of the American Medical Informatics Association following peer review. The version of record is available online at: http://dx.doi.org/10.1093/jamia/ocv008[EN] [Objective] This systematic review aims to identify and compare the existing processes and methodologies that have been published in the literature for defining clinical information models (CIMs) that support the semantic interoperability of electronic health record (EHR) systems. [Material and Methods] Following the preferred reporting items for systematic reviews and meta-analyses systematic review methodology, the authors reviewed published papers between 2000 and 2013 that covered that semantic interoperability of EHRs, found by searching the PubMed, IEEE Xplore, and ScienceDirect databases. Additionally, after selection of a final group of articles, an inductive content analysis was done to summarize the steps and methodologies followed in order to build CIMs described in those articles. [Results] Three hundred and seventy-eight articles were screened and thirty six were selected for full review. The articles selected for full review were analyzed to extract relevant information for the analysis and characterized according to the steps the authors had followed for clinical information modeling. [Discussion] Most of the reviewed papers lack a detailed description of the modeling methodologies used to create CIMs. A representative example is the lack of description related to the definition of terminology bindings and the publication of the generated models. However, this systematic review confirms that most clinical information modeling activities follow very similar steps for the definition of CIMs. Having a robust and shared methodology could improve their correctness, reliability, and quality. [Conclusion] Independently of implementation technologies and standards, it is possible to find common patterns in methods for developing CIMs, suggesting the viability of defining a unified good practice methodology to be used by any clinical information modeler.This research has been partially funded by the Instituto de Salud Carlos III (Platform for Innovation in Medical Technologies and Health), grant PT13/0006/0036 and the Spanish Ministry of Economy and Competitiveness, grants TIN2010-21388-C02-01 and PTQ-12-05620.Moreno-Conde, A.; Moner Cano, D.; Da Cruz, WD.; Santos, MR.; Maldonado Segura, JA.; Robles Viejo, M.; Kalra, D. (2015). Clinical information modeling processes for semantic interoperability of electronic health records: systematic review and inductive analysis. Journal of the American Medical Informatics Association. 22(4):925-934. https://doi.org/10.1093/jamia/ocv008S925934224Goossen W Goossen-Baremans A van der Zel M . Detailed clinical models: a review. Healthc Inform Res. 2010;16:201.Beeler, G. W. (1998). HL7 Version 3—An object-oriented methodology for collaborative standards development1Presented at the International Medical Informatics Association Working Group 16 Conference on Standardisation in Medical Informatics—Towards International Consensus and Cooperation, Bermuda, 12 September, 1997.1. International Journal of Medical Informatics, 48(1-3), 151-161. doi:10.1016/s1386-5056(97)00121-4Dolin, R. H., Alschuler, L., Boyer, S., Beebe, C., Behlen, F. M., Biron, P. V., & Shabo (Shvo), A. (2006). HL7 Clinical Document Architecture, Release 2. Journal of the American Medical Informatics Association, 13(1), 30-39. doi:10.1197/jamia.m1888Fast Health Interoperability Resources (FHIR). http://www.hl7.org/fhir/. Accessed July 18, 2014.Beale T . Archetypes: constraint-based domain models for futureproof information systems. OOPSLA 2002 Workshop Behav Semant. 2002.ISO 13606:2008 - Health informatics - Electronic health record communication. 2008. www.iso.org/iso/catalogue_detail.htm?csnumber=40784.OpenEHR. http://www.openehr.org/. Accessed July 18, 2014.Clinical Information Modeling Initiative (CIMI). http://www.opencimi.org/. Accessed July 18, 2014.Goossen WT . Using detailed clinical models to bridge the gap between clinicians and HIT. Stud Health Technol Inf. 2008;141:3–10.Oniki TA Coyle JF Parker CG . Lessons learned in detailed clinical modeling at Intermountain Healthcare. J Am Med Inform Assoc. 2014. Advance access published; doi:10.1136/amiajnl-2014-002875.Jacobson I Booch G Rumbaugh J . The Unified Software Development Process. Massachusetts, USA: Addison-Wesley Reading; 1999.Ahn, S., Huff, S. M., Kim, Y., & Kalra, D. (2013). Quality metrics for detailed clinical models. International Journal of Medical Informatics, 82(5), 408-417. doi:10.1016/j.ijmedinf.2012.09.006Kalra D . Editorial principles for the development of standards for the structure and content of health records. 2012. https://www.rcplondon.ac.uk/sites/default/files/documents/editorial-principles-for-the-development-of-record-standards.pdf. Accessed July 16, 2014.Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ, 339(jul21 1), b2535-b2535. doi:10.1136/bmj.b2535Transparent reporting of systematic reviews and meta-analyses (PRISMA). http://www.prisma-statement.org/. Accessed July 18, 2014.US National Library of Medicine. http://www.pubmed.gov. Accessed July 18, 2014.IEEE Xplore Digital Library. http://ieeexplore.ieee.org/. Accessed July 18, 2014.ScienceDirect. http://www.sciencedirect.com/. Accessed July 18, 2014.Elo, S., & Kyngäs, H. (2008). The qualitative content analysis process. Journal of Advanced Nursing, 62(1), 107-115. doi:10.1111/j.1365-2648.2007.04569.xRinner C Kohler M Hübner-Bloder G . Creating ISO/EN 13606 archetypes based on clinical information needs. In: Proceedings of EFMI Special Topic Conference, 14–15 April 2011, Lǎsko, Slovenia e-Health Across Borders Without Boundaries. 2011:14–15.Muñoz Carrero A Romero Gutiérrez A Marco Cuenca G . Manual práctico de interoperabilidad semántica para entornos sanitarios basada en arquetipos. Unidad de investigación en Telemedicina y e-Salud. Instituto de Salud Carlos III - Ministerio de Economía y Competitividad. 2013.Kalra D . Editorial principles for the development of standards for the structure and content of health records. 2012. https://www.rcplondon.ac.uk/sites/default/files/documents/editorial-principles-for-the-development-of-record-standards.pdf . Accessed July 18, 2015.Yuksel, M., & Dogac, A. (2011). Interoperability of Medical Device Information and the Clinical Applications: An HL7 RMIM based on the ISO/IEEE 11073 DIM. IEEE Transactions on Information Technology in Biomedicine, 15(4), 557-566. doi:10.1109/titb.2011.2151868Nagy M Hanzlicek P Precková P . Semantic interoperability in Czech healthcare environment supported by HL7 version 3. Methods Inf Med. 2010;49:186.LOPEZ, D., & BLOBEL, B. (2009). A development framework for semantically interoperable health information systems. International Journal of Medical Informatics, 78(2), 83-103. doi:10.1016/j.ijmedinf.2008.05.009Lopez DM Blobel B . Enhanced semantic interoperability by profiling health informatics standards. Methods Inf Med. 2009;48:170–177.Lopez DM Blobel B . Enhanced semantic interpretability by healthcare standards profiling. Stud Health Technol Inform. 2008;136:735.Knaup, P., Garde, S., & Haux, R. (2007). Systematic planning of patient records for cooperative care and multicenter research. International Journal of Medical Informatics, 76(2-3), 109-117. doi:10.1016/j.ijmedinf.2006.08.002Goossen, W. T. F., Ozbolt, J. G., Coenen, A., Park, H.-A., Mead, C., Ehnfors, M., & Marin, H. F. (2004). Development of a Provisional Domain Model for the Nursing Process for Use within the Health Level 7 Reference Information Model. Journal of the American Medical Informatics Association, 11(3), 186-194. doi:10.1197/jamia.m1085Anderson, H. V., Weintraub, W. S., Radford, M. J., Kremers, M. S., Roe, M. T., Shaw, R. E., … Tcheng, J. E. (2013). Standardized Cardiovascular Data for Clinical Research, Registries, and Patient Care. Journal of the American College of Cardiology, 61(18), 1835-1846. doi:10.1016/j.jacc.2012.12.047Jian, W.-S., Hsu, C.-Y., Hao, T.-H., Wen, H.-C., Hsu, M.-H., Lee, Y.-L., … Chang, P. (2007). Building a portable data and information interoperability infrastructure—framework for a standard Taiwan Electronic Medical Record Template. Computer Methods and Programs in Biomedicine, 88(2), 102-111. doi:10.1016/j.cmpb.2007.07.014Spigolon, D. N., & Moro, C. M. C. (2012). Arquétipos do conjunto de dados essenciais de enfermagem para atendimento de portadoras de endometriose. Revista Gaúcha de Enfermagem, 33(4), 22-32. doi:10.1590/s1983-14472012000400003Späth, M. B., & Grimson, J. (2011). Applying the archetype approach to the database of a biobank information management system. International Journal of Medical Informatics, 80(3), 205-226. doi:10.1016/j.ijmedinf.2010.11.002Smith, K., & Kalra, D. (2008). Electronic health records in complementary and alternative medicine. International Journal of Medical Informatics, 77(9), 576-588. doi:10.1016/j.ijmedinf.2007.11.005Bax, M. P., Kalra, D., & Santos, M. R. (2012). Dealing with the Archetypes Development Process for a Regional EHR System. Applied Clinical Informatics, 03(03), 258-275. doi:10.4338/aci-2011-12-ra-0074Moner D Moreno A Maldonado JA . Using archetypes for defining CDA templates. Stud Health Technol Inform. 2012;180:53–57.Moner D Maldonado JA Boscá D . CEN EN13606 normalisation framework implementation experiences. In: Seamless Care, Safe Care: The Challenges of Interoperability and Patient Safety in Health Care: Proceedings of the EFMI Special Topic Conference, June 2–4, 2010; Reykjavik, Iceland. IOS Press; 2010: 136.Marcos, M., Maldonado, J. A., Martínez-Salvador, B., Boscá, D., & Robles, M. (2013). Interoperability of clinical decision-support systems and electronic health records using archetypes: A case study in clinical trial eligibility. Journal of Biomedical Informatics, 46(4), 676-689. doi:10.1016/j.jbi.2013.05.004Leslie H . International developments in openEHR archetypes and templates. Health Inf Manag J. 2008;37:38.Kohl CD Garde S Knaup P . Facilitating secondary use of medical data by using openEHR archetypes. Stud Health Technol Inform. 2009;160:1117–1121.Garde, S., Hovenga, E., Buck, J., & Knaup, P. (2007). Expressing clinical data sets with openEHR archetypes: A solid basis for ubiquitous computing. International Journal of Medical Informatics, 76, S334-S341. doi:10.1016/j.ijmedinf.2007.02.004Garcia D Moro CM Cicogna PE . Method to integrate clinical guidelines into the electronic health record (EHR) by applying the archetypes approach. Stud Health Technol Inform. 2012;192:871–875.Duftschmid, G., Rinner, C., Kohler, M., Huebner-Bloder, G., Saboor, S., & Ammenwerth, E. (2013). The EHR-ARCHE project: Satisfying clinical information needs in a Shared Electronic Health Record System based on IHE XDS and Archetypes. International Journal of Medical Informatics, 82(12), 1195-1207. doi:10.1016/j.ijmedinf.2013.08.002Dias, R. D., Cook, T. W., & Freire, S. M. (2011). Modeling healthcare authorization and claim submissions using the openEHR dual-model approach. BMC Medical Informatics and Decision Making, 11(1). doi:10.1186/1472-6947-11-60Buck, J., Garde, S., Kohl, C. D., & Knaup-Gregori, P. (2009). Towards a comprehensive electronic patient record to support an innovative individual care concept for premature infants using the openEHR approach. International Journal of Medical Informatics, 78(8), 521-531. doi:10.1016/j.ijmedinf.2009.03.001Puentes, J., Roux, M., Montagner, J., & Lecornu, L. (2012). Development framework for a patient-centered record. Computer Methods and Programs in Biomedicine, 108(3), 1036-1051. doi:10.1016/j.cmpb.2012.06.007Liu, D., Wang, X., Pan, F., Yang, P., Xu, Y., Tang, X., … Rao, K. (2010). Harmonization of health data at national level: A pilot study in China. International Journal of Medical Informatics, 79(6), 450-458. doi:10.1016/j.ijmedinf.2010.03.002Liu, D., Wang, X., Pan, F., Xu, Y., Yang, P., & Rao, K. (2008). Web-based infectious disease reporting using XML forms. International Journal of Medical Informatics, 77(9), 630-640. doi:10.1016/j.ijmedinf.2007.10.011Kim, Y., & Park, H.-A. (2011). Development and Validation of Detailed Clinical Models for Nursing Problems in Perinatal care. Applied Clinical Informatics, 02(02), 225-239. doi:10.4338/aci-2011-01-ra-0007Khan, W. A., Hussain, M., Afzal, M., Amin, M. B., Saleem, M. A., & Lee, S. (2013). Personalized-Detailed Clinical Model for Data Interoperability Among Clinical Standards. Telemedicine and e-Health, 19(8), 632-642. doi:10.1089/tmj.2012.0189Jing, X., Kay, S., Marley, T., Hardiker, N. R., & Cimino, J. J. (2012). Incorporating personalized gene sequence variants, molecular genetics knowledge, and health knowledge into an EHR prototype based on the Continuity of Care Record standard. Journal of Biomedical Informatics, 45(1), 82-92. doi:10.1016/j.jbi.2011.09.001Hsu, W., Taira, R. K., El-Saden, S., Kangarloo, H., & Bui, A. A. T. (2012). Context-Based Electronic Health Record: Toward Patient Specific Healthcare. IEEE Transactions on Information Technology in Biomedicine, 16(2), 228-234. doi:10.1109/titb.2012.2186149Hoy D Hardiker NR McNicoll IT . Collaborative development of clinical templates as a national resource. Int J Med Inf. 2009;78:S3–S8.Buyl, R., & Nyssen, M. (2009). Structured electronic physiotherapy records. International Journal of Medical Informatics, 78(7), 473-481. doi:10.1016/j.ijmedinf.2009.02.007D’Amore JD Mandel JC Kreda DA . Are Meaningful Use Stage 2 certified EHRs ready for interoperability? Findings from the SMART C-CDA Collaborative. J Am Med Inform Assoc. 2014. Advance access published; doi:10.1136/amiajnl-2014-002883.Kalra D Tapuria A Austin T . Quality requirements for EHR archetypes. In: MIE; 2012: 48–52.Garde S Hovenga EJ Gränz J . Managing archetypes for sustainable and semantically interoperable electronic health records. Electron J Health Inform. 2007;2:e9.Madsen M Leslie H Hovenga EJS . Sustainable clinical knowledge management: an archetype development life cycle. Stud Health Technol Inform. 2010;151:115–132.Kohl CD Garde S Knaup P . Facilitating the openEHR approach-organizational structures for defining high-quality archetypes. Stud Health Technol Inform. 2008;136:437.Stroetmann VN Kalra D Lewalle P . Semantic interoperability for better health and safer healthcare. European Commission, Directorate-General Information Society and Media; 2009. http://dx.doi.org/10.2759/38514

Validating archetypes for the Multiple Sclerosis Functional Composite

Background Numerous information models for electronic health records, such as openEHR archetypes are available. The quality of such clinical models is important to guarantee standardised semantics and to facilitate their interoperability. However, validation aspects are not regarded sufficiently yet. The objective of this report is to investigate the feasibility of archetype development and its community-based validation process, presuming that this review process is a practical way to ensure high-quality information models amending the formal reference model definitions. Methods A standard archetype development approach was applied on a case set of three clinical tests for multiple sclerosis assessment: After an analysis of the tests, the obtained data elements were organised and structured. The appropriate archetype class was selected and the data elements were implemented in an iterative refinement process. Clinical and information modelling experts validated the models in a structured review process. Results Four new archetypes were developed and publicly deployed in the openEHR Clinical Knowledge Manager, an online platform provided by the openEHR Foundation. Afterwards, these four archetypes were validated by domain experts in a team review. The review was a formalised process, organised in the Clinical Knowledge Manager. Both, development and review process turned out to be time- consuming tasks, mostly due to difficult selection processes between alternative modelling approaches. The archetype review was a straightforward team process with the goal to validate archetypes pragmatically. Conclusions The quality of medical information models is crucial to guarantee standardised semantic representation in order to improve interoperability. The validation process is a practical way to better harmonise models that diverge due to necessary flexibility left open by the underlying formal reference model definitions. This case study provides evidence that both community- and tool- enabled review processes, structured in the Clinical Knowledge Manager, ensure archetype quality. It offers a pragmatic but feasible way to reduce variation in the representation of clinical information models towards a more unified and interoperable model

Archetype Modeling Methodology

[EN] Clinical Information Models (CIMs) expressed as archetypes play an essential role in the design and development of current Electronic Health Record (EHR) information structures. Although there exist many experiences about using archetypes in the literature, a comprehensive and formal methodology for archetype modeling does not exist. Having a modeling methodology is essential to develop quality archetypes, in order to guide the development of EHR systems and to allow the semantic interoperability of health data. In this work, an archetype modeling methodology is proposed. This paper describes its phases, the inputs and outputs of each phase, and the involved participants and tools. It also includes the description of the possible strategies to organize the modeling process. The proposed methodology is inspired by existing best practices of CIMs, software and ontology development. The methodology has been applied and evaluated in regional and national EHR projects. The application of the methodology provided useful feedback and improvements, and confirmed its advantages. The conclusion of this work is that having a formal methodology for archetype development facilitates the definition and adoption of interoperable archetypes, improves their quality, and facilitates their reuse among different information systems and EHR projects. Moreover, the proposed methodology can be also a reference for CIMs development using any other formalism.This work was partially funded by grant DI-14-06564 (Doctorados Industriales) of the Ministerio de Economia y Competitividad of Spain. The authors would also thank the participants of all R&D projects that have served to evaluate and improve the presented methodology.Moner Cano, D.; Maldonado Segura, JA.; Robles Viejo, M. (2018). Archetype Modeling Methodology. Journal of Biomedical Informatics. 79:71-81. https://doi.org/10.1016/j.jbi.2018.02.003S71817

Quality framework for semantic interoperability in health informatics: definition and implementation

Aligned with the increased adoption of Electronic Health Record (EHR) systems, it is recognized that semantic interoperability provides benefits for promoting patient safety and continuity of care. This thesis proposes a framework of quality metrics and recommendations for developing semantic interoperability resources specially focused on clinical information models, which are defined as formal specifications of structure and semantics for representing EHR information for a specific domain or use case. This research started with an exploratory stage that performed a systematic literature review with an international survey about the clinical information modelling best practice and barriers. The results obtained were used to define a set of quality models that were validated through Delphi study methodologies and end user survey, and also compared with related quality standards in those areas that standardization bodies had a related work programme. According to the obtained research results, the defined framework is based in the following models: Development process quality model: evaluates the alignment with the best practice in clinical information modelling and defines metrics for evaluating the tools applied as part of this process. Product quality model: evaluates the semantic interoperability capabilities of clinical information models based on the defined meta-data, data elements and terminology bindings. Quality in use model: evaluates the suitability of adopting semantic interoperability resources by end users in their local projects and organisations. Finally, the quality in use model was implemented within the European Interoperability Asset register developed by the EXPAND project with the aim of applying this quality model in a broader scope to contain any relevant material for guiding the definition, development and implementation of interoperable eHealth systems in our continent. Several European projects already expressed interest in using the register, which will now be sustained by the European Institute for Innovation through Health Data

Patientenübergreifende, multiple Verwendung von Patientendaten für die klinische Forschung unter Nutzung von Archetypen

Sowohl in der Routineversorgung als auch in klinischen Studien werden immer mehr Daten elektronisch verarbeitet. Trotzdem ist ein Austausch von Daten zwischen beiden Bereichen häufig noch nicht etabliert. Dies führt dazu, dass Daten mehrfach erfasst werden müssen. Die redundante Datenerfassung ist zeitaufwändig und kann zu Inkonsistenzen zwischen Krankenhausinformationssystem (KIS) und Studiendatenmanagementsystem (SDMS) führen. Obwohl ein Datenaustausch zwischen Forschung und Versorgung oft technisch möglich wäre, scheitert er meist noch an mangelnder semantischer Interoperabilität. Archetypen sind ein innovatives Konzept zur Gestaltung von flexiblen und leicht erweiterbaren elektronischen Gesundheitsakten. Sie ermöglichen semantische Interopera-bilität zwischen Systemen, welche dieselben Archetypen nutzen. Das Archetypen-Konzept hat mittlerweile auch Eingang in internationale Standards gefunden (ISO 13606). Die openEHR-Spezifikationen definieren ein mit ISO 13606 kompatibles jedoch weiter-gehendes Modell für elektronische Gesundheitsakten. Bisher wurden Archetypen hauptsächlich für Informationssysteme in der Routineversorgung und weniger für die klinische Forschung entwickelt und genutzt. Ziel dieser Arbeit war es daher, basierend auf den openEHR-Spezifikationen und Archetypen generische Ansätze zu erarbeiten, die eine multiple Verwendung von Daten aus der Versorgung in der Forschung ermöglichen und deren Umsetzbarkeit zu prüfen. In einer Voruntersuchung wurde ermittelt, dass 35 % der in der betrachteten Studie zu erhebenden Merkmalsarten aus dem untersuchten KIS übernommen werden könnten, wenn die Daten dort elektronisch und ausreichend strukturiert vorlägen. In einem zweiten Schritt wurde mit openSDMS der Prototyp eines auf Archetypen basierenden integrierten elektronischen Gesundheitsakten- und Studiendatenmanagementsystems zur Verfügung gestellt. Aus der Voruntersuchung und der Implementierung von openSDMS wurden Anforderungen abgeleitet und eine auf openEHR-Archetypen basierende Referenzarchitektur entwickelt, welche die Nutzung von Daten aus KIS in klinischen Studien unterstützt. Dabei wird sowohl die Integration von KIS beschrieben, die auf Archetypen basieren, als auch von klassischen KIS. Kernkomponenten dieser Architektur sind auf Archetypen basierende semantische Annotationen von Studiendaten sowie Import- und Exportmodule, welche die Archetype Query Language nutzen. Die vorgestellte Referenzarchitektur ermöglicht den Übergang von der multiplen Erfassung hin zur multiplen Verwendung von Daten in Forschung und Versorgung. Um die entwickelte Referenzarchitektur realisieren zu können, werden geeignete Archetypen auch für Forschungsdaten benötigt. Daher wurden Archetypen zur Dokumentation aller Datenelemente der vier CDASH Domänen ‚Common Identifier Variables‘, ‚Common Timing Variables‘, ‚Adverse Events‘ sowie ‚Prior and Concomitant Medications‘ spezifiziert (Studiendaten). Hierzu wurden insgesamt 23 Merkmalsarten basierend auf Archetypen neu definiert, wozu drei bestehende Archetypen spezialisiert und zwei neu entwickelt wurden. Zur Definition von CDASH-konformen elektronischen Datenerhebungsbogen für die betrachteten Domänen wurden, basierend auf den spezifizierten Archetypen, vier openEHR-Templates entworfen. Ferner wurden 71 Merkmalsarten in 16 Archetypen zur Dokumentation von Studien-Metadaten definiert. Alle neu entworfenen Archetypen wurden jeweils in englischer und deutscher Sprache beschrieben und können nun als Referenzinformationsmodell für Forschungsdaten genutzt werden. Ergänzend wurden alle von den bereitgestellten Archetypen definierten Merkmalsarten auf die im Bereich der klinischen Forschung etablierten Modelle BRIDG, CDASH und ODM abgebildet

Usability analysis of contending electronic health record systems

In this paper, we report measured usability of two leading EHR systems during procurement. A total of 18 users participated in paired-usability testing of three scenarios: ordering and managing medications by an outpatient physician, medicine administration by an inpatient nurse and scheduling of appointments by nursing staff. Data for audio, screen capture, satisfaction rating, task success and errors made was collected during testing. We found a clear difference between the systems for percentage of successfully completed tasks, two different satisfaction measures and perceived learnability when looking at the results over all scenarios. We conclude that usability should be evaluated during procurement and the difference in usability between systems could be revealed even with fewer measures than were used in our study. © 2019 American Psychological Association Inc. All rights reserved.Peer reviewe