Online Guide for Electronic Health Evaluation Approaches: Systematic Scoping Review and Concept Mapping Study

BACKGROUND: Despite the increase in use and high expectations of digital health solutions, scientific evidence about the effectiveness of electronic health (eHealth) and other aspects such as usability and accuracy is lagging behind. eHealth solutions are complex interventions, which require a wide array of evaluation approaches that are capable of answering the many different questions that arise during the consecutive study phases of eHealth development and implementation. However, evaluators seem to struggle in choosing suitable evaluation approaches in relation to a specific study phase. OBJECTIVE: The objective of this project was to provide a structured overview of the existing eHealth evaluation approaches, with the aim of assisting eHealth evaluators in selecting a suitable approach for evaluating their eHealth solution at a specific evaluation study phase. METHODS: Three consecutive steps were followed. Step 1 was a systematic scoping review, summarizing existing eHealth evaluation approaches. Step 2 was a concept mapping study asking eHealth researchers about approaches for evaluating eHealth. In step 3, the results of step 1 and 2 were used to develop an "eHealth evaluation cycle" and subsequently compose the online "eHealth methodology guide." RESULTS: The scoping review yielded 57 articles describing 50 unique evaluation approaches. The concept mapping study questioned 43 eHealth researchers, resulting in 48 unique approaches. After removing duplicates, 75 unique evaluation approaches remained. Thereafter, an "eHealth evaluation cycle" was developed, consisting of six evaluation study phases: conceptual and planning, design, development and usability, pilot (feasibility), effectiveness (impact), uptake (implementation), and all phases. Finally, the "eHealth methodology guide" was composed by assigning the 75 evaluation approaches to the specific study phases of the "eHealth evaluation cycle." CONCLUSIONS: Seventy-five unique evaluation approaches were found in the literature and suggested by eHealth researchers, which served as content for the online "eHealth methodology guide." By assisting evaluators in selecting a suitable evaluation approach in relation to a specific study phase of the "eHealth evaluation cycle," the guide aims to enhance the quality, safety, and successful long-term implementation of novel eHealth solutions

Cardiac rehabilitation and survival in a large representative community cohort of Dutch patients

To assess the effects of multi-disciplinary cardiac rehabilitation (CR) on survival in the full population of patients with an acute coronary syndrome (ACS) and patients that underwent coronary revascularization and/or heart valve surgery. Population-based cohort study in the Netherlands using insurance claims database covering ∼22% of the Dutch population (3.3 million persons). All patients with an ACS with or without ST elevation, and patients who underwent coronary revascularization and/or valve surgery in the period 2007-10 were included. Patients were categorized as having received CR when an insurance claim for CR was made within the first 180 days after the cardiac event or revascularization. The primary outcome was survival time from the inclusion date, limited to a total follow-up period of 4 years, with a minimum of 180 days. Propensity score weighting was used to control for confounding by indication. Among 35 919 patients with an ACS and/or coronary revascularization or valve surgery, 11 014 (30.7%) received CR. After propensity score weighting, the adjusted hazard ratio (HR) associated with receiving CR was 0.65 (95% CI 0.56-0.77). The largest benefit was observed for patients who underwent coronary artery bypass grafting (CABG) and/or valve surgery (HR = 0.55, 95% CI 0.42-0.74). In a large and representative community cohort of Dutch patients with an ACS and/or intervention, CR was associated with a substantial survival benefit up to 4 years. This survival benefit was present regardless of age, type of diagnosis, and type of interventio

Effect of a web-based audit and feedback intervention with outreach visits on the clinical performance of multidisciplinary teams: a cluster-randomized trial in cardiac rehabilitation

BACKGROUND: The objective of this study was to assess the effect of a web-based audit and feedback (A&amp;F) intervention with outreach visits to support decision-making by multidisciplinary teams.METHODS: We performed a multicentre cluster-randomized trial within the field of comprehensive cardiac rehabilitation (CR) in the Netherlands. Our participants were multidisciplinary teams in Dutch CR centres who were enrolled in the study between July 2012 and December 2013 and received the intervention for at least 1 year. The intervention included web-based A&amp;F with feedback on clinical performance, facilities for goal setting and action planning, and educational outreach visits. Teams were randomized either to receive feedback that was limited to psychosocial rehabilitation (study group A) or to physical rehabilitation (study group B). The main outcome measure was the difference in performance between study groups in 11 care processes and six patient outcomes, measured at patient level. Secondary outcomes included effects on guideline concordance for the four main CR therapies.RESULTS: Data from 18 centres (14,847 patients) were analysed, of which 12 centres (9353 patients) were assigned to group A and six (5494 patients) to group B. During the intervention, a total of 233 quality improvement goals was identified by participating teams, of which 49 (21%) were achieved during the study period. Except for a modest improvement in data completeness (4.5% improvement per year; 95% CI 0.65 to 8.36), we found no effect of our intervention on any of our primary or secondary outcome measures.CONCLUSIONS: Within a multidisciplinary setting, our web-based A&amp;F intervention engaged teams to define local performance improvement goals but failed to support them in actually completing the improvement actions that were needed to achieve those goals. Future research should focus on improving the actionability of feedback on clinical performance and on addressing the socio-technical perspective of the implementation process.TRIAL REGISTRATION: NTR3251.</p

How to use concept mapping to identify barriers and facilitators of an electronic quality improvement intervention

Systematic quality improvement (QI) interventions are increasingly used to change complex health care systems. Results of randomized clinical trials can provide quantitative evidence whether QI interventions were effective but they do not teach us why and how QI was (not) achieved. Qualitative research methods can answer these questions but typically involve only a small group of respondents against high resources. Concept mapping methodology overcomes these drawbacks by integrating results from qualitative group sessions with multivariate statistical analysis to represent ideas of diverse stakeholders visually on maps in an efficient way. This paper aims to describe how to use concept mapping to qualitatively gain insight into barriers and facilitators of an electronic QI intervention and presents experiences with the method from an ongoing case study to evaluate a QI system in the field of cardiac rehabilitation in the Netherland

The effect of computerized decision support on barriers to guideline implementation: A qualitative study in outpatient cardiac rehabilitation

Context: Computerized decision support systems (CDSSs) can be used to improve the implementation of clinical practice guidelines by changing the behaviour of care professionals. While the influence of system characteristics on the effectiveness of CDSSs is studied, little is known about the relation between cognitive, organizational and environmental factors, and CDSSs' effectiveness. Objective: To assess the effect of CDSSs on cognitive, organizational, and environmental factors that hamper guideline implementation. Design: In-depth, semi-structured interviews with care professionals, on reasons for improved adherence or persistent non-adherence to the prevailing guideline after successful adoption of a CDSS. All remarks regarding guideline implementation were extracted and classified using the conceptual framework from Cabana et al. [5]. Setting: Outpatient cardiac rehabilitation clinics. Participants: Care professionals that used the CARDSS decision support system for therapeutic decision making in cardiac rehabilitation. Results: Twenty-nine rehabilitation nurses and physiotherapists from 21 Dutch clinics were interviewed. CARDSS improved guideline adherence by increasing its users' familiarity with the guidelines' recommendations and decision logic, by overcoming users' inertia to previous practice, and by reducing guideline complexity for example by facilitating calculation and interpretation of data. If the system's recommendations were shared with patients, refusal to participate in therapies reduced. CARDSS never incited users to target barriers related to organizational or environmental constraints. Conclusion: Our results suggest that computerized decision support can improve guideline implementation by increasing the knowledge of preferred practice, by reducing inertia to previous practice, and by reducing guideline complexity. However, computerized decision support is not effective when organizational or procedural changes are required that users consider to be beyond their tasks and responsibilities. (C) 2010 Elsevier Ireland Ltd. All rights reserve