The EHR-ARCHE project: Satisfying clinical information needs in a Shared Electronic Health Record System based on IHE XDS and Archetypes

AbstractPurposeWhile contributing to an improved continuity of care, Shared Electronic Health Record (EHR) systems may also lead to information overload of healthcare providers. Document-oriented architectures, such as the commonly employed IHE XDS profile, which only support information retrieval at the level of documents, are particularly susceptible for this problem. The objective of the EHR-ARCHE project was to develop a methodology and a prototype to efficiently satisfy healthcare providers’ information needs when accessing a patient's Shared EHR during a treatment situation. We especially aimed to investigate whether this objective can be reached by integrating EHR Archetypes into an IHE XDS environment.MethodsUsing methodical triangulation, we first analysed the information needs of healthcare providers, focusing on the treatment of diabetes patients as an exemplary application domain. We then designed ISO/EN 13606 Archetypes covering the identified information needs. To support a content-based search for fine-grained information items within EHR documents, we extended the IHE XDS environment with two additional actors. Finally, we conducted a formative and summative evaluation of our approach within a controlled study.ResultsWe identified 446 frequently needed diabetes-specific information items, representing typical information needs of healthcare providers. We then created 128 Archetypes and 120 EHR documents for two fictive patients. All seven diabetes experts, who evaluated our approach, preferred the content-based search to a conventional XDS search. Success rates of finding relevant information was higher for the content-based search (100% versus 80%) and the latter was also more time-efficient (8–14min versus 20min or more).ConclusionsOur results show that for an efficient satisfaction of health care providers’ information needs, a content-based search that rests upon the integration of Archetypes into an IHE XDS-based Shared EHR system is superior to a conventional metadata-based XDS search

Quantification of the resilience of primary care networks by stress testing the health care system

There are practically no quantitative tools for understanding how much stress a health care system can absorb before it loses its ability to provide care. We propose to measure the resilience of health care systems with respect to changes in the density of primary care providers. We develop a computational model on a 1-to-1 scale for a countrywide primary care sector based on patient-sharing networks. Nodes represent all primary care providers in a country; links indicate patient flows between them. The removal of providers could cause a cascade of patient displacements, as patients have to find alternative providers. The model is calibrated with nationwide data from Austria that includes almost all primary care contacts over 2 y. We assign 2 properties to every provider: the “CareRank” measures the average number of displacements caused by a provider’s removal (systemic risk) as well as the fraction of patients a provider can absorb when others default (systemic benefit). Below a critical number of providers, large-scale cascades of patient displacements occur, and no more providers can be found in a given region. We quantify regional resilience as the maximum fraction of providers that can be removed before cascading events prevent coverage for all patients within a district. We find considerable regional heterogeneity in the critical transition point from resilient to nonresilient behavior. We demonstrate that health care resilience cannot be quantified by physician density alone but must take into account how networked systems respond and restructure in response to shocks. The approach can identify systemically relevant providers

Improving the informational continuity of care in diabetes mellitus treatment with a nationwide Shared EHR system: Estimates from Austrian claims data

Purpose Shared Electronic Health Record (EHR) systems, which provide a health information exchange (HIE) within a community of care, were found to be a key enabler of informational continuity of diabetes mellitus (DM) care. Quantitative analyses of the actual contribution of Shared EHR systems to informational continuity of care are rare. The goal of this study was to quantitatively analyze (i) the degree of fragmentation of DM care in Austria as an indicator for the need for HIE, and (ii) the quantity of information (i.e. number of documents) from Austrian DM patients that would be made available by a nationwide Shared EHR system for HIE. Methods Our analyses are based on social security claims data of 7.9 million Austrians from 2006 and 2007. DM patients were identified through medication data and inpatient diagnoses. The degree of fragmentation was determined by the number of different healthcare providers per patient. The amount of information that would be made available by a nationwide Shared EHR system was estimated by the number of documents that would have been available to a healthcare provider if he had access to information on the patient’s visits to any of the other healthcare providers. As a reference value we determined the number of locally available documents that would have originated from the patient’s visits to the healthcare provider himself. We performed our analysis for two types of systems: (i) a “comprehensive” Shared EHR system (SEHRS), where each visit of a patient results in a single document (progress note), and (ii) the Austrian ELGA system, which allows four specific document types to be shared. Results 391,630 DM patients were identified, corresponding to 4.7% of the Austrian population. More than 90% of the patients received health services from more than one healthcare provider in one year. Both, the SEHRS as well as ELGA would have multiplied the available information during a patient visit in comparison to an isolated local EHR system; the median ratio of external to local medical documents was between 1:1 for a typical visit at a primary care provider (SEHRS as well as ELGA) and 39:1 (SEHRS) respectively 28:1 (ELGA) for a typical visit at a hospital. Conclusions Due to the high degree of care fragmentation, there is an obvious need for HIE for Austrian DM patients. Both, the SEHRS as well as ELGA could provide a substantial contribution to informational continuity of care in Austrian DM treatment. Hospitals and specialists would have gained the most amount of external information, primary care providers and pharmacies would have at least doubled their available information. Despite being the most important potential feeders of a national Shared EHR system according to our analysis, primary care providers will not tap their full corresponding potential under the current implementation scenario of ELGA

Semantic validation of standard based electronic health record documents with W3C XML Schema

++43 1 40400 6693 (Phone) [email protected] 2 Summary Objectives. The goal of this article is to examine whether W3C XML Schema provides a practicable solution for the semantic validation of standard based electronic health record (EHR) documents. With semantic validation we mean that the EHR documents are checked for conformance with the underlying archetypes and reference model. Methods. We describe an approach that allows XML Schemas to be derived from archetypes based on a specific naming convention. The archetype constraints are augmented with additional components of the reference model within the XML Schema representation. A copy of the EHR document that is transformed according to the before-mentioned naming convention is used for the actual validation against the XML Schema. Results. We tested our approach by semantically validating EHR documents conformant to three different ISO/EN 13606 archetypes respective to three sections of the CDA implementation guide "Continuity of Care Document (CCD)" and an implementation guide for diabetes therapy data. We further developed a tool to automate the different steps of our semantic validation approach. Conclusions. For two particular kinds of archetype prescriptions, individual transformations are required for the corresponding EHR documents. Otherwise, a fully generic validation is possible. In general, we consider W3C XML Schema as a practicable solution for the semantic validation of standard based EHR documents

Reply to Klitz and Niklasson: Can viral infections explain the cross-sectional Austrian diabetes data?

The vast majority of diabetes patients suffer from type II diabetes. There are several theories for potential causes of its development, the most favored being the "exhaustion" of beta-cells by chronic excess caloric intake combined with inactivity leading to obesity, diabetes, and cardiovascular disease. Viral infection could be a potential cause, however mainly for type I diabetes. Furthermore, autoimmunity might play a role in type I patients..

How sick is Austria? – A decision support framework for different evaluations of the burden of disease within the Austrian population based on different data sources

ABSTRACT Objectives In healthcare it is crucial to have a fundamental knowledge of the burden of diseases within the population. Therefore we aimed to develop an Atlas of Epidemiology to gain better insight on the epidemiological situation. Based on primary and secondary health care data, we aimed to present results in interactive charts and maps, comprehensible to experts and the general public. The atlas builds a framework for rapid deployment of new data and results in a reproducible and efficient way. As a first use case three methods based on two different databases for the estimation of diabetes prevalence in Austria are compared. Approach Datasources: (i) reimbursement data 2006/2007 (GAP-DRG); (ii) national routine health survey (ATHIS) for 2006/2007. Methods for diabetes prevalence estimation: 1) ATC-ICD statistically relates pseudonymized data on medications to data on diagnoses from hospitalizations and sick leaves. 2) With the method Experts, medical experts assign specific medications to diabetes diagnoses. Patients with these medications are identified together with hospitalized diabetes diagnosed patients in GAP-DRG. 3) In ATHIS a sample of 15.000 persons was questioned if they a) ever had diabetes and b) were treated against diabetes in the last 12 months. Results are projected onto the Austrian population. Patients are divided by 10-year age-classes, gender and state. For the publicly online framework, implemented in html and javascript, pre-processed data in different granularity is required and used. Results Maps of Austria represent the prevalence of diabetes for each method and granularity level. The difference of the methods can be seen by clicking on the next map. For different age-classes (resp. different gender) the three methods can be compared directly within a bar chart. The technology for a rapid deployment of new data is now developed. For the use case first results have already been presented to decision makers, and feedback has been incorporated. Conclusion Besides depicting disease prevalence, the atlas of epidemiology also allows to visualize health care service data and results of simulation models in a fast and efficient way, which is important for decision makers. Soon the results of the ATC-ICD project on the prevalence of different diseases based on ICD9 diagnoses and medication data will be published in an aggregated form. This project is part of the K-Project dexhelpp in COMET – Competence Centers for Excellent Technologies that is funded by BMVIT, BMWGJ and transacted by FFG

Clinical information modeling processes for semantic interoperability of electronic health records: systematic review and inductive analysis

This is a pre-copyedited, author-produced PDF of an article accepted for publication in Journal of the American Medical Informatics Association following peer review. The version of record is available online at: http://dx.doi.org/10.1093/jamia/ocv008[EN] [Objective] This systematic review aims to identify and compare the existing processes and methodologies that have been published in the literature for defining clinical information models (CIMs) that support the semantic interoperability of electronic health record (EHR) systems. [Material and Methods] Following the preferred reporting items for systematic reviews and meta-analyses systematic review methodology, the authors reviewed published papers between 2000 and 2013 that covered that semantic interoperability of EHRs, found by searching the PubMed, IEEE Xplore, and ScienceDirect databases. Additionally, after selection of a final group of articles, an inductive content analysis was done to summarize the steps and methodologies followed in order to build CIMs described in those articles. [Results] Three hundred and seventy-eight articles were screened and thirty six were selected for full review. The articles selected for full review were analyzed to extract relevant information for the analysis and characterized according to the steps the authors had followed for clinical information modeling. [Discussion] Most of the reviewed papers lack a detailed description of the modeling methodologies used to create CIMs. A representative example is the lack of description related to the definition of terminology bindings and the publication of the generated models. However, this systematic review confirms that most clinical information modeling activities follow very similar steps for the definition of CIMs. Having a robust and shared methodology could improve their correctness, reliability, and quality. [Conclusion] Independently of implementation technologies and standards, it is possible to find common patterns in methods for developing CIMs, suggesting the viability of defining a unified good practice methodology to be used by any clinical information modeler.This research has been partially funded by the Instituto de Salud Carlos III (Platform for Innovation in Medical Technologies and Health), grant PT13/0006/0036 and the Spanish Ministry of Economy and Competitiveness, grants TIN2010-21388-C02-01 and PTQ-12-05620.Moreno-Conde, A.; Moner Cano, D.; Da Cruz, WD.; Santos, MR.; Maldonado Segura, JA.; Robles Viejo, M.; Kalra, D. (2015). Clinical information modeling processes for semantic interoperability of electronic health records: systematic review and inductive analysis. Journal of the American Medical Informatics Association. 22(4):925-934. https://doi.org/10.1093/jamia/ocv008S925934224Goossen W Goossen-Baremans A van der Zel M . Detailed clinical models: a review. Healthc Inform Res. 2010;16:201.Beeler, G. W. (1998). HL7 Version 3—An object-oriented methodology for collaborative standards development1Presented at the International Medical Informatics Association Working Group 16 Conference on Standardisation in Medical Informatics—Towards International Consensus and Cooperation, Bermuda, 12 September, 1997.1. International Journal of Medical Informatics, 48(1-3), 151-161. doi:10.1016/s1386-5056(97)00121-4Dolin, R. H., Alschuler, L., Boyer, S., Beebe, C., Behlen, F. M., Biron, P. V., & Shabo (Shvo), A. (2006). HL7 Clinical Document Architecture, Release 2. Journal of the American Medical Informatics Association, 13(1), 30-39. doi:10.1197/jamia.m1888Fast Health Interoperability Resources (FHIR). http://www.hl7.org/fhir/. Accessed July 18, 2014.Beale T . Archetypes: constraint-based domain models for futureproof information systems. OOPSLA 2002 Workshop Behav Semant. 2002.ISO 13606:2008 - Health informatics - Electronic health record communication. 2008. www.iso.org/iso/catalogue_detail.htm?csnumber=40784.OpenEHR. http://www.openehr.org/. Accessed July 18, 2014.Clinical Information Modeling Initiative (CIMI). http://www.opencimi.org/. Accessed July 18, 2014.Goossen WT . Using detailed clinical models to bridge the gap between clinicians and HIT. Stud Health Technol Inf. 2008;141:3–10.Oniki TA Coyle JF Parker CG . Lessons learned in detailed clinical modeling at Intermountain Healthcare. J Am Med Inform Assoc. 2014. Advance access published; doi:10.1136/amiajnl-2014-002875.Jacobson I Booch G Rumbaugh J . The Unified Software Development Process. Massachusetts, USA: Addison-Wesley Reading; 1999.Ahn, S., Huff, S. M., Kim, Y., & Kalra, D. (2013). Quality metrics for detailed clinical models. International Journal of Medical Informatics, 82(5), 408-417. doi:10.1016/j.ijmedinf.2012.09.006Kalra D . Editorial principles for the development of standards for the structure and content of health records. 2012. https://www.rcplondon.ac.uk/sites/default/files/documents/editorial-principles-for-the-development-of-record-standards.pdf. Accessed July 16, 2014.Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ, 339(jul21 1), b2535-b2535. doi:10.1136/bmj.b2535Transparent reporting of systematic reviews and meta-analyses (PRISMA). http://www.prisma-statement.org/. Accessed July 18, 2014.US National Library of Medicine. http://www.pubmed.gov. Accessed July 18, 2014.IEEE Xplore Digital Library. http://ieeexplore.ieee.org/. Accessed July 18, 2014.ScienceDirect. http://www.sciencedirect.com/. Accessed July 18, 2014.Elo, S., & Kyngäs, H. (2008). The qualitative content analysis process. Journal of Advanced Nursing, 62(1), 107-115. doi:10.1111/j.1365-2648.2007.04569.xRinner C Kohler M Hübner-Bloder G . Creating ISO/EN 13606 archetypes based on clinical information needs. In: Proceedings of EFMI Special Topic Conference, 14–15 April 2011, Lǎsko, Slovenia e-Health Across Borders Without Boundaries. 2011:14–15.Muñoz Carrero A Romero Gutiérrez A Marco Cuenca G . Manual práctico de interoperabilidad semántica para entornos sanitarios basada en arquetipos. Unidad de investigación en Telemedicina y e-Salud. Instituto de Salud Carlos III - Ministerio de Economía y Competitividad. 2013.Kalra D . Editorial principles for the development of standards for the structure and content of health records. 2012. https://www.rcplondon.ac.uk/sites/default/files/documents/editorial-principles-for-the-development-of-record-standards.pdf . Accessed July 18, 2015.Yuksel, M., & Dogac, A. (2011). Interoperability of Medical Device Information and the Clinical Applications: An HL7 RMIM based on the ISO/IEEE 11073 DIM. IEEE Transactions on Information Technology in Biomedicine, 15(4), 557-566. doi:10.1109/titb.2011.2151868Nagy M Hanzlicek P Precková P . Semantic interoperability in Czech healthcare environment supported by HL7 version 3. Methods Inf Med. 2010;49:186.LOPEZ, D., & BLOBEL, B. (2009). A development framework for semantically interoperable health information systems. International Journal of Medical Informatics, 78(2), 83-103. doi:10.1016/j.ijmedinf.2008.05.009Lopez DM Blobel B . Enhanced semantic interoperability by profiling health informatics standards. Methods Inf Med. 2009;48:170–177.Lopez DM Blobel B . Enhanced semantic interpretability by healthcare standards profiling. Stud Health Technol Inform. 2008;136:735.Knaup, P., Garde, S., & Haux, R. (2007). Systematic planning of patient records for cooperative care and multicenter research. International Journal of Medical Informatics, 76(2-3), 109-117. doi:10.1016/j.ijmedinf.2006.08.002Goossen, W. T. F., Ozbolt, J. G., Coenen, A., Park, H.-A., Mead, C., Ehnfors, M., & Marin, H. F. (2004). Development of a Provisional Domain Model for the Nursing Process for Use within the Health Level 7 Reference Information Model. Journal of the American Medical Informatics Association, 11(3), 186-194. doi:10.1197/jamia.m1085Anderson, H. V., Weintraub, W. S., Radford, M. J., Kremers, M. S., Roe, M. T., Shaw, R. E., … Tcheng, J. E. (2013). Standardized Cardiovascular Data for Clinical Research, Registries, and Patient Care. Journal of the American College of Cardiology, 61(18), 1835-1846. doi:10.1016/j.jacc.2012.12.047Jian, W.-S., Hsu, C.-Y., Hao, T.-H., Wen, H.-C., Hsu, M.-H., Lee, Y.-L., … Chang, P. (2007). Building a portable data and information interoperability infrastructure—framework for a standard Taiwan Electronic Medical Record Template. Computer Methods and Programs in Biomedicine, 88(2), 102-111. doi:10.1016/j.cmpb.2007.07.014Spigolon, D. N., & Moro, C. M. C. (2012). Arquétipos do conjunto de dados essenciais de enfermagem para atendimento de portadoras de endometriose. Revista Gaúcha de Enfermagem, 33(4), 22-32. doi:10.1590/s1983-14472012000400003Späth, M. B., & Grimson, J. (2011). Applying the archetype approach to the database of a biobank information management system. International Journal of Medical Informatics, 80(3), 205-226. doi:10.1016/j.ijmedinf.2010.11.002Smith, K., & Kalra, D. (2008). Electronic health records in complementary and alternative medicine. International Journal of Medical Informatics, 77(9), 576-588. doi:10.1016/j.ijmedinf.2007.11.005Bax, M. P., Kalra, D., & Santos, M. R. (2012). Dealing with the Archetypes Development Process for a Regional EHR System. Applied Clinical Informatics, 03(03), 258-275. doi:10.4338/aci-2011-12-ra-0074Moner D Moreno A Maldonado JA . Using archetypes for defining CDA templates. Stud Health Technol Inform. 2012;180:53–57.Moner D Maldonado JA Boscá D . CEN EN13606 normalisation framework implementation experiences. In: Seamless Care, Safe Care: The Challenges of Interoperability and Patient Safety in Health Care: Proceedings of the EFMI Special Topic Conference, June 2–4, 2010; Reykjavik, Iceland. IOS Press; 2010: 136.Marcos, M., Maldonado, J. A., Martínez-Salvador, B., Boscá, D., & Robles, M. (2013). Interoperability of clinical decision-support systems and electronic health records using archetypes: A case study in clinical trial eligibility. Journal of Biomedical Informatics, 46(4), 676-689. doi:10.1016/j.jbi.2013.05.004Leslie H . International developments in openEHR archetypes and templates. Health Inf Manag J. 2008;37:38.Kohl CD Garde S Knaup P . Facilitating secondary use of medical data by using openEHR archetypes. Stud Health Technol Inform. 2009;160:1117–1121.Garde, S., Hovenga, E., Buck, J., & Knaup, P. (2007). Expressing clinical data sets with openEHR archetypes: A solid basis for ubiquitous computing. International Journal of Medical Informatics, 76, S334-S341. doi:10.1016/j.ijmedinf.2007.02.004Garcia D Moro CM Cicogna PE . Method to integrate clinical guidelines into the electronic health record (EHR) by applying the archetypes approach. Stud Health Technol Inform. 2012;192:871–875.Duftschmid, G., Rinner, C., Kohler, M., Huebner-Bloder, G., Saboor, S., & Ammenwerth, E. (2013). The EHR-ARCHE project: Satisfying clinical information needs in a Shared Electronic Health Record System based on IHE XDS and Archetypes. International Journal of Medical Informatics, 82(12), 1195-1207. doi:10.1016/j.ijmedinf.2013.08.002Dias, R. D., Cook, T. W., & Freire, S. M. (2011). Modeling healthcare authorization and claim submissions using the openEHR dual-model approach. BMC Medical Informatics and Decision Making, 11(1). doi:10.1186/1472-6947-11-60Buck, J., Garde, S., Kohl, C. D., & Knaup-Gregori, P. (2009). Towards a comprehensive electronic patient record to support an innovative individual care concept for premature infants using the openEHR approach. International Journal of Medical Informatics, 78(8), 521-531. doi:10.1016/j.ijmedinf.2009.03.001Puentes, J., Roux, M., Montagner, J., & Lecornu, L. (2012). Development framework for a patient-centered record. Computer Methods and Programs in Biomedicine, 108(3), 1036-1051. doi:10.1016/j.cmpb.2012.06.007Liu, D., Wang, X., Pan, F., Yang, P., Xu, Y., Tang, X., … Rao, K. (2010). Harmonization of health data at national level: A pilot study in China. International Journal of Medical Informatics, 79(6), 450-458. doi:10.1016/j.ijmedinf.2010.03.002Liu, D., Wang, X., Pan, F., Xu, Y., Yang, P., & Rao, K. (2008). Web-based infectious disease reporting using XML forms. International Journal of Medical Informatics, 77(9), 630-640. doi:10.1016/j.ijmedinf.2007.10.011Kim, Y., & Park, H.-A. (2011). Development and Validation of Detailed Clinical Models for Nursing Problems in Perinatal care. Applied Clinical Informatics, 02(02), 225-239. doi:10.4338/aci-2011-01-ra-0007Khan, W. A., Hussain, M., Afzal, M., Amin, M. B., Saleem, M. A., & Lee, S. (2013). Personalized-Detailed Clinical Model for Data Interoperability Among Clinical Standards. Telemedicine and e-Health, 19(8), 632-642. doi:10.1089/tmj.2012.0189Jing, X., Kay, S., Marley, T., Hardiker, N. R., & Cimino, J. J. (2012). Incorporating personalized gene sequence variants, molecular genetics knowledge, and health knowledge into an EHR prototype based on the Continuity of Care Record standard. Journal of Biomedical Informatics, 45(1), 82-92. doi:10.1016/j.jbi.2011.09.001Hsu, W., Taira, R. K., El-Saden, S., Kangarloo, H., & Bui, A. A. T. (2012). Context-Based Electronic Health Record: Toward Patient Specific Healthcare. IEEE Transactions on Information Technology in Biomedicine, 16(2), 228-234. doi:10.1109/titb.2012.2186149Hoy D Hardiker NR McNicoll IT . Collaborative development of clinical templates as a national resource. Int J Med Inf. 2009;78:S3–S8.Buyl, R., & Nyssen, M. (2009). Structured electronic physiotherapy records. International Journal of Medical Informatics, 78(7), 473-481. doi:10.1016/j.ijmedinf.2009.02.007D’Amore JD Mandel JC Kreda DA . Are Meaningful Use Stage 2 certified EHRs ready for interoperability? Findings from the SMART C-CDA Collaborative. J Am Med Inform Assoc. 2014. Advance access published; doi:10.1136/amiajnl-2014-002883.Kalra D Tapuria A Austin T . Quality requirements for EHR archetypes. In: MIE; 2012: 48–52.Garde S Hovenga EJ Gränz J . Managing archetypes for sustainable and semantically interoperable electronic health records. Electron J Health Inform. 2007;2:e9.Madsen M Leslie H Hovenga EJS . Sustainable clinical knowledge management: an archetype development life cycle. Stud Health Technol Inform. 2010;151:115–132.Kohl CD Garde S Knaup P . Facilitating the openEHR approach-organizational structures for defining high-quality archetypes. Stud Health Technol Inform. 2008;136:437.Stroetmann VN Kalra D Lewalle P . Semantic interoperability for better health and safer healthcare. European Commission, Directorate-General Information Society and Media; 2009. http://dx.doi.org/10.2759/38514

Acquiring Guideline-enabled data driven clinical knowledge model using formally verified refined knowledge acquisition method

Background and Objective: Validation and verification are the critical requirements for the knowledge acquisition method of the clinical decision support system (CDSS). After acquiring the medical knowledge from diverse sources, the rigorous validation and formal verification process are required before creating the final knowledge model. Previously, we have proposed a hybrid knowledge acquisition method with the support of a rigorous validation process for acquiring medical knowledge from clinical practice guidelines (CPGs) and patient data for the treatment of oral cavity cancer. However, due to lack of formal verification process, it involves various inconsistencies in knowledge relevant to the formalism of knowledge, conformance to CPGs, quality of knowledge, and complexities of knowledge acquisition artifacts.Methods: This paper presents the refined knowledge acquisition (ReKA) method, which uses the Z formal verification process. The ReKA method adopts the verification method and explores the mechanism of theorem proving using the Z notation. It enhances a hybrid knowledge acquisition method to thwart the inconsistencies using formal verification.Results: ReKA adds a set of nine additional criteria to be used to have a final valid refined clinical knowledge model. These criteria ensure the validity of the final knowledge model concerning formalism of knowledge, conformance to GPGs, quality of the knowledge, usage of stringent conditions and treatment plans, and inconsistencies possibly resulting from the complexities. Evaluation, using four medical knowledge acquisition scenarios, shows that newly added knowledge in CDSS due to the additional criteria by the ReKA method always produces a valid knowledge model. The final knowledge model was also evaluated with 1229 oral cavity patient cases, which outperformed with an accuracy of 72.57% compared to a similar approach with an accuracy of 69.7%. Furthermore, the ReKA method identified a set of decision paths (about 47.8%) in the existing approach, which results in a final knowledge model with low quality, non-conformed from standard CPGs.Conclusion: ReKA refined the hybrid knowledge acquisition method by discovering the missing steps in the current validation process at the acquisition stage. As a formally proven method, it always yields a valid knowledge model having high quality, supporting local practices, and influenced by standard CPGs. Furthermore, the final knowledge model obtained from ReKA also preserves the performance such as the accuracy of the individual source knowledge models.N/