How to identify when a performance indicator has run its course

The official published version can be found at the link below.Increasing numbers of countries are using indicators to evaluate the quality of clinical care, with some linking payment to achievement. For performance frameworks to remain effective the indicators need to be regularly reviewed. The frameworks cannot cover all clinical areas, and achievement on chosen indicators will eventually reach a ceiling beyond which further improvement is not feasible. However, there has been little work on how to select indictors for replacement. The Department of Health decided in 2008 that it would regularly replace indicators in the national primary care pay for performance scheme, the Quality and Outcomes Framework, making a rigorous approach to removal a priority. We draw on our previous work on pay for performance and our current work advising the National Institute for Health and Clinical Excellence (NICE) on the Quality and Outcomes Framework to suggest what should be considered when planning to remove indicators from a clinical performance framework

Measurement tools and process indicators of patient safety culture in primary care. A mixed methods study by the LINNEAUS collaboration on patient safety in primary care.

This is the author accepted manuscript. The final version is available from the publisher via the DOI in this recordBACKGROUND: There is little guidance available to healthcare practitioners about what tools they might use to assess the patient safety culture. OBJECTIVE: To identify useful tools for assessing patient safety culture in primary care organizations in Europe; to identify those aspects of performance that should be assessed when investigating the relationship between safety culture and performance in primary care. METHODS: Two consensus-based studies were carried out, in which subject matter experts and primary healthcare professionals from several EU states rated (a) the applicability to their healthcare system of several existing safety culture assessment tools and (b) the appropriateness and usefulness of a range of potential indicators of a positive patient safety culture to primary care settings. The safety culture tools were field-tested in four countries to ascertain any challenges and issues arising when used in primary care. RESULTS: The two existing tools that received the most favourable ratings were the Manchester patient safety framework (MaPsAF primary care version) and the Agency for healthcare research and quality survey (medical office version). Several potential safety culture process indicators were identified. The one that emerged as offering the best combination of appropriateness and usefulness related to the collection of data on adverse patient events. CONCLUSION: Two tools, one quantitative and one qualitative, were identified as applicable and useful in assessing patient safety culture in primary care settings in Europe. Safety culture indicators in primary care should focus on the processes rather than the outcomes of care.The research leading to these results has received funding from the European Community’s Seventh Framework Programme FP7/2008 – 2012 under grant agreement no. 223424

Identifying patient and practice characteristics associated with patient-reported experiences of safety problems and harm: a cross-sectional study using a multilevel modelling approach.

This is the author accepted manuscript. The final version is available from BMJ Publishing Group via the DOI in this record.OBJECTIVE: To identify patient and family practice characteristics associated with patient-reported experiences of safety problems and harm. DESIGN: Cross-sectional study combining data from the individual postal administration of the validated Patient Reported Experiences and Outcomes of Safety in Primary Care (PREOS-PC) questionnaire to a random sample of patients in family practices (response rate=18.4%) and practice-level data for those practices obtained from NHS Digital. We built linear multilevel multivariate regression models to model the association between patient-level (clinical and sociodemographic) and practice-level (size and case-mix, human resources, indicators of quality and safety of care, and practice safety activation) characteristics, and outcome measures. SETTING: Practices distributed across five regions in the North, Centre and South of England. PARTICIPANTS: 1190 patients registered in 45 practices purposefully sampled (maximal variation in practice size and levels of deprivation). MAIN OUTCOME MEASURES: Self-reported safety problems, harm and overall perception of safety. RESULTS: Higher self-reported levels of safety problems were associated with younger age of patients (beta coefficient 0.15) and lower levels of practice safety activation (0.44). Higher self-reported levels of harm were associated with younger age (0.13) and worse self-reported health status (0.23). Lower self-reported healthcare safety was associated with lower levels of practice safety activation (0.40). The fully adjusted models explained 4.5% of the variance in experiences of safety problems, 8.6% of the variance in harm and 4.4% of the variance in perceptions of patient safety. CONCLUSIONS: Practices' safety activation levels and patients' age and health status are associated with patient-reported safety outcomes in English family practices. The development of interventions aimed at improving patient safety outcomes would benefit from focusing on the identified groups.This research is part-funded by the National Institute for Health Research School for Primary Care Research (NIHR SPCR). The views expressed are those of the authors and not necessarily those of the NIHR, the NHS or the Department of Health

Health data processes. A framework for analysing and discussing efficient use and reuse of health data with focus on Patient Reported Outcome (PRO) measures

This is is the final version. Available from Journal of Medical Internet Research via the DOI in this record.The collection and use of patient health data is central to any kind of activity in the healthcare system. This data may be produced during routine clinical processes or obtained directly from the patient using patient-reported outcome (PRO) measures. Although efficiency and other reasons justify data availability for a range of potential relevant uses, these data are nearly always collected for a single specific purpose. The healthcare data literature reflects this narrow scope, and there is limited literature on the joint use of health data for daily clinical use, clinical research, surveillance and administrative purposes. The aim of this paper is to provide a framework for a discussion of the efficient use of health data with specific focus on the role of PRO measures. PRO data may be used: i) at an individual patient level to inform patient care or shared-decision making and tailor care to individual needs or ii) at group level as a complement to health record data e.g. on mortality and readmission to inform service delivery and measure real-world effectiveness of treatment. PRO may be used either for their own sake, to provide valuable information from the patient perspective, or as proxy for clinical data that would be otherwise not feasible to collect. We introduce a framework to analyse any health care activity that involves health data. The framework consists of four data processes (patient identification, data collection, data aggregation and data use), further structured into two dichotomous dimensions in each data process (level: group vs patient; and timeframe: ad hoc vs systematic). This framework is used to analyse various health activities with respect to joint use of data considering the technical, legal, organisational and logistical challenges that characterize each data process. Finally, we propose a model for joint use of health data with data collected during follow-up as basis. Demands for health data will continue to increase which will further add to the need for the concerted use and reuse of PRO data for parallel purposes. Repeated and uncoordinated PRO data collection for the same patient for different purposes results in misuse of resources for the healthcare system as well as reduced response rates owing to questionnaire fatigue. PRO data can be routinely collected both at the hospital (in- as well as outpatients) and outside of hospital settings, in primary or social care settings, or in the patient’s home provided the health informatics infrastructure is in place. In the future, clinical settings are likely to be a prominent source of PRO data; however we are also likely to see increased remote collection of PRO data by patients in their own home (telePRO). Data collection for research and quality surveillance will have to adapt to this circumstance and adopt complementary data capture methods which take advantage of the utility of PRO data collected during daily clinical practice. The European Union’s regulation with respect to the protection of personal data, General Data Protection Regulation, imposes severe restrictions on use of health data for parallel purposes and steps should be taken to alleviate the consequences while still protecting personal data against misuse.National Institute for Health Research (NIHR

Withdrawing performance indicators: retrospective analysis of general practice performance under UK Quality and Outcomes Framework.

This is a freely-available open access publication. Please cite the published version which is available via the DOI link in this record.OBJECTIVES: To investigate the effect of withdrawing incentives on recorded quality of care, in the context of the UK Quality and Outcomes Framework pay for performance scheme. DESIGN: Retrospective longitudinal study. SETTING: Data for 644 general practices, from 2004/05 to 2011/12, extracted from the Clinical Practice Research Datalink. PARTICIPANTS: All patients registered with any of the practices over the study period-13,772,992 in total. INTERVENTION: Removal of financial incentives for aspects of care for patients with asthma, coronary heart disease, diabetes, stroke, and psychosis. MAIN OUTCOME MEASURES: Performance on eight clinical quality indicators withdrawn from a national incentive scheme: influenza immunisation (asthma) and lithium treatment monitoring (psychosis), removed in April 2006; blood pressure monitoring (coronary heart disease, diabetes, stroke), cholesterol concentration monitoring (coronary heart disease, diabetes), and blood glucose monitoring (diabetes), removed in April 2011. Multilevel mixed effects multiple linear regression models were used to quantify the effect of incentive withdrawal. RESULTS: Mean levels of performance were generally stable after the removal of the incentives, in both the short and long term. For the two indicators removed in April 2006, levels in 2011/12 were very close to 2005/06 levels, although a small but statistically significant drop was estimated for influenza immunisation. For five of the six indicators withdrawn from April 2011, no significant effect on performance was seen following removal and differences between predicted and observed scores were small. Performance on related outcome indicators retained in the scheme (such as blood pressure control) was generally unaffected. CONCLUSIONS: Following the removal of incentives, levels of performance across a range of clinical activities generally remained stable. This indicates that health benefits from incentive schemes can potentially be increased by periodically replacing existing indicators with new indicators relating to alternative aspects of care. However, all aspects of care investigated remained indirectly or partly incentivised in other indicators, and further work is needed to assess the generalisability of the findings when incentives are fully withdrawn.National Institute for Health Research (NIHR

An overview of self-administered health literacy instruments

This is the final version. Available from Public Library of Science via the DOI in this record.With the increasing recognition of health literacy as a worldwide research priority, the development and refinement of indices to measure the construct is an important area of inquiry. Furthermore, the proliferation of online resources and research means that there is a growing need for self-administered instruments. We undertook a systematic overview to identify all published self-administered health literacy assessment indices to report their content and considerations associated with their administration. A primary aim of this study was to assist those seeking to employ a self-reported health literacy index to select one that has been developed and validated for an appropriate context, as well as with desired administration characteristics. Systematic searches were carried out in four electronic databases, and studies were included if they reported the development and/or validation of a novel health literacy assessment measure. Data were systematically extracted on key characteristics of the instruments: breadth of construct ("generic" vs. "contentor context- specific" health literacy), whether it was an original instrument or a derivative, country of origin, administration characteristics, age of target population (adult vs. pediatric), and evidence for validity. 35 articles met the inclusion criteria. There were 27 original instruments (27/35; 77.1%) and 8 derivative instruments (8/35; 22.9%). 22 indices measured "general" health literacy (22/35; 62.9%) while the remainder measured condition- or context- specific health literacy (13/35; 37.1%). Most health literacy measures were developed in the United States (22/35; 62.9%), and about half had adequate face, content, and construct validity (16/35; 45.7%). Given the number of measures available for many specific conditions and contexts, and that several have acceptable validity, our findings suggest that the research agenda should shift towards the investigation and elaboration of health literacy as a construct itself, in order for research in health literacy measurement to progress.National Institute for Health Research (NIHR

Measuring patient safety in primary care: The development and validation of the “Patient Reported Experiences and Outcomes of Safety in Primary Care” (PREOS-PC)

This is the author accepted manuscript. The final version is available from Annals of Family Medicine via the DOI in this recordPURPOSE We set out to develop and validate a patient-reported instrument for measuring experiences and outcomes related to patient safety in primary care. METHOD The instrument was developed in a multistage process supported by an international expert panel and informed by a systematic review of instruments, a meta-synthesis of qualitative studies, 4 patient focus groups, 18 cognitive interviews, and a pilot study. The trial version of Patient Reported Experiences and Outcomes of Safety in Primary Care (PREOS-PC) covered 5 domains and 11 scales: practice activation (1 scale); patient activation (1 scale); experiences of patient safety events (1 scale); harm (6 scales); and general perceptions of patient safety (2 scales). The questionnaire was posted to 6,736 patients in 45 practices across England. We used “gold standard” psychometric methods to evaluate its acceptability, reliability, structural and construct validity, and ability to discriminate among practices. RESULTS 1,244 completed questionnaires (18.5%) were returned. Median itemspecific response rate was 91.3% (interquartile range 28.0%). No major ceiling or floor effects were observed. All 6 multi-item scales showed high internal consistency (Cronbach’s α 0.75-0.96). Factor analysis, correlation between scales, and known group analyses generally supported structural and construct validity. The scales demonstrated a heterogeneous ability to discriminate between practices. The final version of PREOS-PC consisted of 5 domains, 8 scales, and 58 items. CONCLUSIONS PREOS-PC is a new multi-dimensional patient safety instrument for primary care developed with experts and patients. Initial testing shows its potential for use in primary care, and future developments will further address its use in actual clinical practice.UK National Institute for Health Research School for Primary Care Research (NIHR SPCR

Effect of financial incentives on incentivised and non-incentivised clinical activities: longitudinal analysis of data from the UK Quality and Outcomes Framework

This is the final version. Available on open access from BMJ Publishing Group via the DOI in this recordData sharing: Technical appendix and statistical code available from the corresponding author ([email protected]). The dataset was derived from the General Practice Research Database and is not available from the authors, but it can be derived on application to GPRD.OBJECTIVE: To investigate whether the incentive scheme for UK general practitioners led them to neglect activities not included in the scheme. DESIGN: Longitudinal analysis of achievement rates for 42 activities (23 included in incentive scheme, 19 not included) selected from 428 identified indicators of quality of care. SETTING: 148 general practices in England (653 500 patients). MAIN OUTCOME MEASURES: Achievement rates projected from trends in the pre-incentive period (2000-1 to 2002-3) and actual rates in the first three years of the scheme (2004-5 to 2006-7). RESULTS: Achievement rates improved for most indicators in the pre-incentive period. There were significant increases in the rate of improvement in the first year of the incentive scheme (2004-5) for 22 of the 23 incentivised indicators. Achievement for these indicators reached a plateau after 2004-5, but quality of care in 2006-7 remained higher than that predicted by pre-incentive trends for 14 incentivised indicators. There was no overall effect on the rate of improvement for non-incentivised indicators in the first year of the scheme, but by 2006-7 achievement rates were significantly below those predicted by pre-incentive trends. CONCLUSIONS: There were substantial improvements in quality for all indicators between 2001 and 2007. Improvements associated with financial incentives seem to have been achieved at the expense of small detrimental effects on aspects of care that were not incentivised.There was no direct funding for this study, but the National Primary Care Research and Development Centre receives core funding from the UK Department of Health

A research agenda on patient safety in primary care. Recommendations by the LINNEAUS collaboration on patient safety in primary care

This is the final version of the article. Available from Taylor & Francis via the DOI in this record.BACKGROUND: Healthcare can cause avoidable serious harm to patients. Primary care is not an exception, and the relative lack of research in this area lends urgency to a better understanding of patient safety, the future research agenda and the development of primary care oriented safety programmes. OBJECTIVE: To outline a research agenda for patient safety improvement in primary care in Europe and beyond. METHODS: The LINNEAUS collaboration partners analysed existing research on epidemiology and classification of errors, diagnostic and medication errors, safety culture, and learning for and improving patient safety. We discussed ideas for future research in several meetings, workshops and congresses with LINNEAUS collaboration partners, practising GPs, researchers in this field, and policy makers. RESULTS: This paper summarizes and integrates the outcomes of the LINNEAUS collaboration on patient safety in primary care. It proposes a research agenda on improvement strategies for patient safety in primary care. In addition, it provides background information to help to connect research in this field with practicing GPs and other healthcare workers in primary care. CONCLUSION: Future research studies should target specific primary care domains, using prospective methods and innovative methods such as patient involvement.The research leading to these results has received funding from the European Community’s Seventh Framework Programme FP7/2008–2012 under grant agreement no. 223424