Prioritising pre-hospital outcome measures with a multi-stakeholder group: a consensus methods study

Context: A consensus event to discuss and prioritise ambulance service care outcome measures was held with 43 participants from a range of professional backgrounds including Commissioners; Policy makers; clinicians; managers; academics and patient and public representatives. Problem: Ambulance services in England manage 8 million emergency calls per years and treat 6.5 million people. Services are currently unable to ascertain whether the care they provide is safe, effective and of good quality as they receive no information about patients once they have been discharged from their care. The lack of robust patient focussed outcome measures for ambulance care means there is no opportunity for identifying and sharing good practice, identifying problems and measuring the impact of service developments and innovations. Assessment of problem and analysis of its causes: Historically ambulance service performance has been measured using response time as a proxy measure for quality. Although the limitations of this measure are recognised there is a lack of consensus on which outcome measures are important and little opportunity to measure alternatives due to poor information on what happens to patients after their ambulance service contact. The PhOEBE NIHR research programme aims to develop a linked ambulance service and secondary care dataset and to assess quality of care in this patient group using outcome measures identified from the literature and in consultation with different stakeholder groups. This means that for the first time the ambulance service will be able to assess the quality of care they provide to patients, rather than just how quickly the ambulance arrived. Intervention: Potential outcome measures identified from 2 systematic reviews were categorised into 1 of 3 headings (Service/operational, patient management and patient outcomes) and participants were pre-allocated to a discussion group. All discussion groups contained participants representing a range of stakeholder view points. Participants took part in small group themed discussions relating to a number of pre-specified outcome measures. They were also able to add to the list of measures. Directly following the discussion participants voted on the importance of the outcome measures in relation to ambulance service care quality. This was done using Turning Point software. Participants rated each outcome measure as either ‘Essential’, ‘Desirable’ or ‘Irrelevant’ using individual key pads. The voting was done independently and anonymously. Real time results were displayed following each vote. Study design: We used an interactive voting system coupled with a modified nominal group technique for the prioritisation of potential ambulance service outcome measures. Strategy for change: Following on from this study the top ranking outcome measures will be further refined as part of a Delphi study, before using the outcome measures to assess ambulance service quality of care in our linked data sample. The methods for linking the ambulance service data to other health care information and the identified outcome measures will enable all UK ambulance services to assess the quality of care they provide to patients and the impact of any service changes on care quality and patient outcomes. Measurement of improvement: The results from the outcome prioritisation voting exercise were ranked based on the highest proportion of ‘Essential’ rated measures. Where over 50% of participants rated a measure as ‘Essential’ these were taken forward and considered in further consensus studies. Effects of changes: From undertaking the consensus event we have prioritised potential ambulance service outcome measures. Lessons learnt: We have established that it is possible to incorporate voting technology into consensus methodologies and provide real time results to participants. Message for others: This research prioritised ambulance service outcome measures. Out of the 40 number of measures considered, the top 5 measures were Accuracy of dispatch decisions; Completeness and accuracy of patient records; Accuracy of call taker identification of different conditions; pain measurement and symptom relief and Patient experience

Developing new ways of measuring the impact of ambulance service care

Background Pre-hospital care in England is provided by ambulance services who deliver a diverse range of services to over 9 million patients a year but there is limited evidence about the effectiveness of this care. Historically ambulance performance has been measured by response times rather than clinical need or effectiveness. Progress on developing more appropriate performance measures is constrained by a lack of information about what happens to patients and their outcome after the pre-hospital component of care. If ambulance service information about patients could be linked to process and outcome data further along the care pathway then relevant measurement tools could be developed that allow a better assessment of the impact of pre-hospital care. The Pre-hospital Outcomes for Evidence Based Evaluation (PhOEBE) project is a 5 year programme of research funded by the UK National Institute of Health Research. Aims & objectives The aim of the programme is to develop new ways of measuring the impact of care provided by the ambulance service to support quality improvement through monitoring, audit and service evaluation. The objectives are to: 1) Review and synthesise the research literature on pre-hospital care outcome measures and identify measures relevant to the NHS and patients for further development; 2) Create a dataset linking routinely collected pre-hospital data, hospital data and mortality data to provide outcome information; 3) Develop new ways of measuring process and outcome indicators including building risk adjustment models that predict the outcomes using the linked data; 4) Explore the practical use of the linked dataset and the risk adjustment models to measure the effectiveness and quality of ambulance service care. Research plans The programme has 4 linked stages; 1. Synthesis of evidence on outcome measures and identification of measures for further development - review and assessment of the evidence base on outcome measurement for pre-hospital care and a consensus studies to identify measures relevant to patients and NHS staff. 2: Linking pre-hospital data with other patient data sources – creating a single dataset that links ambulance service electronic care records with routinely collected Hospital Episode Statistics (HES) and national mortality data. 3. Development of risk adjustment models for outcomes in patients attended by the ambulance service – using the linked data to develop risk adjustment tools that will allow patient differences to be taken into account and differences between expected and actual outcomes to be detected. Particular emphasis will be made to include the broad EMS population and not specific conditions as has been the case in the past. 4. Testing the risk adjustment models to assess if they can be used to measure effectiveness and quality – exploring the practical application of the measures by using them to assess if different ways of providing ambulance service care result in different consequences for patients. Outputs, outcomes and impact The programme will: • Provide a summary of relevant evidence on pre-hospital care outcome measurement • Develop a method for linking healthcare information into a format that can be used to support quality improvement, is acceptable to patients and complies with information legislation • Develop population based models for measuring the impact of pre-hospital care that can be used to monitor quality and safety, evaluate new service innovations and support quality improvement • Provide added value by using routine information and NHS infrastructure to operationalise the process and outcome models so that they will be of use across the NHS Progress to date The programme commenced in June 2011 and ends in May 2016. Two systematic reviews of measures used to measure the impact of ambulance service care (one policy literature and one research literature based) have been completed as has a qualitative study of recent service users to identify aspects of service they value. Potential measures identified by these studies were presented at a consensus conference and then further refined in a Delphi study to prioritise and identify measures for further development. Linked data is currently being created and the next stage will be the development of risk adjusted predictive models for the final identified measures

Prehospital outcomes for ambulance service care: systematic review

Background: Ambulance service performance measurement has previously focused on response times and survival. We conducted a systematic review of the international literature on quality measures and outcomes relating to pre-hospital ambulance service care, aiming to identify a broad range of outcome measures to provide a more meaningful assessment of ambulance service care. Methods: We searched a number of electronic databases including CINAHL, the Cochrane Library, EMBASE, Medline, and Web of Science. For inclusion, studies had to report either research or evaluation conducted in a pre-hospital setting, published in the English language from 1982 to 2011, and reporting either outcome measures or specific outcome instruments. Results: Overall, 181 full-text articles were included: 83 (46%) studies from North America, 50 (28%) from Europe and 21 (12%) from the UK. A total of 176 articles were obtained after examining 257 full-text articles in detail from 5,088 abstracts screened. A further five papers were subsequently identified from references of the articles examined and studies known to the authors. There were 140 articles (77%) which contained at least one survival-related measure, 47 (34%) which included information about length of stay and 87 (48%) which identified at least one place of discharge as an outcome. Limitations: We encountered the problem of incomplete information, for instance studies not specifying which pain scales when these had been used or using survival without a specific time period. Conclusion and recommendations: In addition to measures relating to survival, length of stay and place of discharge, we identified 247 additional outcome measures. Few studies included patient reported or cost outcomes. By identifying a wide range of outcome measures this review will inform further research looking at the feasibility of using a wider range of outcome measures and developing new outcome measures in prehospital research and quality improvement

Should I stay or should I go? - 2:Monitoring influences on NHS staff retention in the post COVID-19 world Winter 2020 to spring 2023

The issue of staff retention in the NHS is not new but has been brought into sharp relief in the post-COVID-19 era of unpreceded staff shortage. While steps have been taken to train new health professionals and recruit from overseas, net gains to the NHS staff complement are at risk of being significantly blunted or defeated in the absence of finding ways to stabilise and enhance the retention of established staff. At the institutional level, what has been widely characterised as a pandemic of early­ exit risks a spiral of inter-related losses becoming endemic. Foreseeable impacts include loss of expertise and institutional memory, degraded capacity to deliver patient care, degraded workforce and work-team stability, loss of return on investment in health professional training, and increased human resource costs to employers (e.g. recruitment and employment of bank/agency staff). All have implications for standards of patient care and the potential to negatively impact on the well-being of staff in-post to the extent that it risks degrading their disposition and/or capacity to remain.The foundation research on which this report is based, 'Should I stay or should I go? NHS staff retention in the post COVID-19 world: Challenges and prospects', was funded by the Economic and Social Research Council, in response to the UKRI open­ call for COVID-19 relevant social research in spring 2020, supplemented by follow-on funding from the health sector. At its inception the research aims were to provide human resource strategy and policy relevant insight into:•the impact of the COVID-19 experiences and its legacy on employees' strength of attachment, commitment and capacity to remain in NHS employment;•the relative salience and strength of push and pull variables on staff stay versus leave intentions and behaviour;•what might need to change to motivate/enable current employees to remain in NHS employment; and•the need, nature and scope for intervention to maintain/enhance retention rates.At Wave Four of our survey, the scope of data gathering broadened from its initial focus on primary impacts arising from COVID-19 in 2020/21 and its legacy to include other features of the post-pandemic work environment, including: staff shortages, workload, job-demands, working conditions, pay and other background climate factors on staff resilience, capacity and disposition to remain.This report provides an overview of headline findings from the NHS employee survey component of our research. The survey sample covered all staff types, however a central focus was on health professionals and associated care staff, due to the more restricted scope for personnel substitution.The current report is focused on findings from Wave Four of the survey, conducted in spring 2023, and represents an update on and point of comparison with findings from the three earlier waves (winter 2020/21, summer/autumn 2021 and spring/summer 2022), published in the University of Bath Institute for Policy Research report series in January 2023 (Weyman et al. 2023).In common with Wave Three, the Wave Four survey was completed by a sample of NHS employees in England. Waves One and Two were completed by UK-wide samples. However, the close alignment of the response profiles across the devolved nation samples in Waves One and Two gives confidence to considering Waves Three and Four findings to have UK-wide generalisability and relevance

Prioritising outcomes measures for ambulance service care: a three stage consensus study

Background Historically ambulance care quality and performance has been measured by response times rather than clinical need or effectiveness. This limits the evidence about the effectiveness of the care the ambulance service provides. The Pre-hospital Outcomes for Evidence Based Evaluation (PhOEBE) project is a 5 year research programme which aims to develop new ways of measuring the performance, quality and impact of ambulance service care that better reflect the care provided. Methods We held a 1 day consensus event with a multi-stakeholder group to identify and prioritise which features of ambulance service care are important and worth further development as measures of performance and quality. The day comprised a series of small group discussions to share opinions about measures identified from literature reviews and the opportunity for participants to add their ideas, together with a live vote to help identify and rank measures for further development. Time measures were considered in a separate online questionnaire. The results from the online questionnaire and Consensus Event were fed into a Delphi study. Results 43 participants attended the consensus event, representing a range of participant groups including patient and public, ambulance services, commissioners and policy groups. The top 5 outcome measures prioritised by the participants were: accuracy of dispatch decisions; completeness and accuracy of patient records; accuracy of call taker identification of different conditions or needs (e.g. heart attack, stroke, suitable for nurse advice); pain measurement and symptom relief; and patient experience. From the online survey of time measures the top 5 measures were time of call to time of arrival at scene; time symptoms start (e.g. chest pain) to time of treatment with balloon for heart attack; time taken to answer the emergency call; total time spent on scene; and incident or start of symptoms to calling the ambulance service. Conclusion Using consensus methods we have identified a set of outcome measures that can potentially be used to measure the performance and quality of ambulance service care. Some of these measures fit with existing government targets, such as response times and time taken to answer the emergency call. However, most measures are new ways of measuring ambulance service care and the importance of these measures and methods of measurement will be assessed as part of an on-going Delphi study

What outcome measures should be developed for pre-hospital care?: results of a consensus event

Background: The PhOEBE (Pre-hospital Outcomes for Evidence Based Evaluation) project is a 5-year research programme which aims to develop new ways of measuring the performance, quality and impact of ambulance service care. As part of this programme we conducted 2 systematic reviews to identify potential measures and held a consensus event to prioritise these measures. Methods: Actual or potential measures or indicators for assessing ambulance service performance or quality of care were identified from two systematic reviews and categorised as clinical management, operational or patient based measures. Time measures were considered separately. We held a consensus event with participants representing clinicians, ambulance operations, commissioners, policy and academic research. Three small group discussion sessions were held and after each session we used turning point software for participants to electronically vote whether they thought each potential measure was essential, desirable or irrelevant. Results: 42 participants took part and discussed and voted on 52 different measures. They could also add measures. The top 5 ranked operational measures were concerned with completeness and accuracy of records; accuracy of triage; appropriateness of service; ambulance training and ambulance utilisation. Top 5 clinical measures were accuracy of dispatch decisions, accuracy of problem identification, compliance with end of life care plans; patient safety and compliance with protocols. The top 5 patient measures were measurement and relief of pain; patient experience; return of spontaneous circulation; complications from care and survival. Conclusions: Accuracy of different types of decision making and compliance with management protocols predominated as essential with pain management the most important patient measure. Management of end of life care was identified by participants. The electronic voting system which provided instant real time feedback was well received by participants. The next stage is a Delphi survey to further refine these measures and include time interval measures

Systematic review of pre-hospital outcomes for evidence-based evaluation of ambulance service care

Background: Ambulance service performance measurement has previously focused on response times and survival (particularly from out-of-hospital cardiac arrest). The PhOEBE (Pre-hospital Outcomes for Evidence Based Evaluation) project is a 5-year research programme which aims to develop new ways of measuring the performance, quality and impact of ambulance service care. As part of this programme we conducted a systematic review of the international literature on quality measures and outcomes relating to pre-hospital ambulance service care, aiming to identify a broad range of outcome measures to provide a more meaningful assessment of ambulance service care. Methods: We searched a number of electronic databases including CINAHL, the Cochrane Library, EMBASE, Medline, and Web of Science. For inclusion, studies had to report either research or evaluation conducted in a pre-hospital setting and published in the English language from 1982 to 2011, reporting either outcome measures or specific outcome instruments. Results: Overall, 181 full-text articles were included: 83 (46%) studies from North America, 50 (28%) from Europe and 21 (12%) from the United Kingdom. Initially, 176 articles were included after examining 257 full-text articles from 5,088 abstracts screened. A further five papers were subsequently identified from references of the articles examined and studies known to the authors. There were 140 articles (77%) which contained at least one survival-related measure, while 47 (26%) included information about length of stay and 87 (48%) identified at least one place of discharge as an outcome. Conclusion: In addition to measures relating to survival, length of stay and place of discharge, we identified over 100 additional outcome measures. Few studies included patient reported outcomes or economic outcomes. By identifying a range of outcome measures, this review will inform the use of a greater range of outcome measures and development of new outcome measures in pre-hospital research and quality improvement

What evidence is there on the effectiveness of different models of delivering urgent care? A rapid review

Objective The purpose of the evidence synthesis is to assess the nature and quality of the existing evidence base on delivery of emergency and urgent care services and identify gaps that require further primary research or evidence synthesis. Methods We have conducted a rapid framework-based evidence synthesis approach. Five separate reviews were conducted linked to themes in the NHS England review. A general and five theme specific database searches were conducted for the years 1995-2014. Relevant systematic reviews and additional primary research papers were included with narrative assessment of evidence quality was conducted for each review. Results The review was completed in six months. In total 45 systematic reviews and 102 primary research studies have been included across all 5 reviews. The key findings for each reviews were 1) Demand - there is little empirical evidence to explain increases in demand for urgent care, 2) Telephone triage - Overall, these services provide , appropriate and safe decision making with high patient satisfaction but required clinical skill mix and effectiveness in a system is unclear , 3) extended paramedic roles have been implemented in various health settings and appear to be successful at reducing transports to hospital, making safe decisions about the need for transport and delivering acceptable, cost-effective care out of hospital. 4)ED – The evidence on co-location of GP services with ED indicates there is potential to improve care. The attempt to summarise the evidence about wider ED operations proved to be too complex and further focused reviews are needed. 5) There is no empirical evidence to support the design and development of urgent care networks. Limitations Although there is a large body of evidence on relevant interventions much of it is weak with only very small numbers of randomised controlled trials identified. Evidence is dominated by single site studies many of which were uncontrolled. Conclusions The evidence gaps of most relevance to the delivery of services are 1) more detailed understanding and mapping of the characteristics of demand to inform service planning, 2) assessment of the current state of urgent care network development and evaluation of effectiveness of different models, and 3) Expanding the current evidence base on existing interventions that are viewed as central to delivery of the NHS England plan by assessing the implications of increasing interventions at scale and measuring costs and system impact. It would be prudent to develop a national picture of existing pilot projects or interventions in development to support decisions about research commissioning

Prioritising outcomes measures for ambulance service care: a three stage consensus study

Background Historically ambulance care quality and performance has been measured by response times rather than clinical need or effectiveness. This limits the evidence about the effectiveness of the care the ambulance service provides. The Pre-hospital Outcomes for Evidence Based Evaluation (PhOEBE) project is a 5 year research programme which aims to develop new ways of measuring the performance, quality and impact of ambulance service care that better reflect the care provided. Methods We held a 1 day consensus event with a multi-stakeholder group to identify and prioritise which features of ambulance service care are important and worth further development as measures of performance and quality. The day comprised a series of small group discussions to share opinions about measures identified from literature reviews and the opportunity for participants to add their ideas, together with a live vote to help identify and rank measures for further development. Time measures were considered in a separate online questionnaire. The results from the online questionnaire and Consensus Event were fed into a Delphi study. Results 43 participants attended the consensus event, representing a range of participant groups including patient and public, ambulance services, commissioners and policy groups. The top 5 outcome measures prioritised by the participants were: Accuracy of dispatch decisions; Completeness and accuracy of patient records; Accuracy of call taker identification of different conditions or needs (e.g. heart attack, stroke, suitable for nurse advice); Pain measurement and symptom relief; Patient experience. From the online survey of time measures the top 5 measures were Time of call to time of arrival at scene; Time symptoms start (e.g. chest pain) to time of treatment with balloon for heart attack; Time taken to answer the emergency call; Total time spent on scene; Incident or start of symptoms to calling the ambulance service Conclusions Using consensus methods we have identified a set of outcome measures that can potentially be used to measure the performance and quality of ambulance service care. Some of these measures fit with existing government targets, such as response times and time taken to answer the emergency call. However most measures are new ways of measuring ambulance service care and the importance of these measures and methods of measurement will be assessed as part of an on-going Delphi study