Prioritising pre-hospital outcome measures with a multi-stakeholder group: a consensus methods study

Context: A consensus event to discuss and prioritise ambulance service care outcome measures was held with 43 participants from a range of professional backgrounds including Commissioners; Policy makers; clinicians; managers; academics and patient and public representatives. Problem: Ambulance services in England manage 8 million emergency calls per years and treat 6.5 million people. Services are currently unable to ascertain whether the care they provide is safe, effective and of good quality as they receive no information about patients once they have been discharged from their care. The lack of robust patient focussed outcome measures for ambulance care means there is no opportunity for identifying and sharing good practice, identifying problems and measuring the impact of service developments and innovations. Assessment of problem and analysis of its causes: Historically ambulance service performance has been measured using response time as a proxy measure for quality. Although the limitations of this measure are recognised there is a lack of consensus on which outcome measures are important and little opportunity to measure alternatives due to poor information on what happens to patients after their ambulance service contact. The PhOEBE NIHR research programme aims to develop a linked ambulance service and secondary care dataset and to assess quality of care in this patient group using outcome measures identified from the literature and in consultation with different stakeholder groups. This means that for the first time the ambulance service will be able to assess the quality of care they provide to patients, rather than just how quickly the ambulance arrived. Intervention: Potential outcome measures identified from 2 systematic reviews were categorised into 1 of 3 headings (Service/operational, patient management and patient outcomes) and participants were pre-allocated to a discussion group. All discussion groups contained participants representing a range of stakeholder view points. Participants took part in small group themed discussions relating to a number of pre-specified outcome measures. They were also able to add to the list of measures. Directly following the discussion participants voted on the importance of the outcome measures in relation to ambulance service care quality. This was done using Turning Point software. Participants rated each outcome measure as either ‘Essential’, ‘Desirable’ or ‘Irrelevant’ using individual key pads. The voting was done independently and anonymously. Real time results were displayed following each vote. Study design: We used an interactive voting system coupled with a modified nominal group technique for the prioritisation of potential ambulance service outcome measures. Strategy for change: Following on from this study the top ranking outcome measures will be further refined as part of a Delphi study, before using the outcome measures to assess ambulance service quality of care in our linked data sample. The methods for linking the ambulance service data to other health care information and the identified outcome measures will enable all UK ambulance services to assess the quality of care they provide to patients and the impact of any service changes on care quality and patient outcomes. Measurement of improvement: The results from the outcome prioritisation voting exercise were ranked based on the highest proportion of ‘Essential’ rated measures. Where over 50% of participants rated a measure as ‘Essential’ these were taken forward and considered in further consensus studies. Effects of changes: From undertaking the consensus event we have prioritised potential ambulance service outcome measures. Lessons learnt: We have established that it is possible to incorporate voting technology into consensus methodologies and provide real time results to participants. Message for others: This research prioritised ambulance service outcome measures. Out of the 40 number of measures considered, the top 5 measures were Accuracy of dispatch decisions; Completeness and accuracy of patient records; Accuracy of call taker identification of different conditions; pain measurement and symptom relief and Patient experience

Developing new ways of measuring the impact of ambulance service care

Background Pre-hospital care in England is provided by ambulance services who deliver a diverse range of services to over 9 million patients a year but there is limited evidence about the effectiveness of this care. Historically ambulance performance has been measured by response times rather than clinical need or effectiveness. Progress on developing more appropriate performance measures is constrained by a lack of information about what happens to patients and their outcome after the pre-hospital component of care. If ambulance service information about patients could be linked to process and outcome data further along the care pathway then relevant measurement tools could be developed that allow a better assessment of the impact of pre-hospital care. The Pre-hospital Outcomes for Evidence Based Evaluation (PhOEBE) project is a 5 year programme of research funded by the UK National Institute of Health Research. Aims & objectives The aim of the programme is to develop new ways of measuring the impact of care provided by the ambulance service to support quality improvement through monitoring, audit and service evaluation. The objectives are to: 1) Review and synthesise the research literature on pre-hospital care outcome measures and identify measures relevant to the NHS and patients for further development; 2) Create a dataset linking routinely collected pre-hospital data, hospital data and mortality data to provide outcome information; 3) Develop new ways of measuring process and outcome indicators including building risk adjustment models that predict the outcomes using the linked data; 4) Explore the practical use of the linked dataset and the risk adjustment models to measure the effectiveness and quality of ambulance service care. Research plans The programme has 4 linked stages; 1. Synthesis of evidence on outcome measures and identification of measures for further development - review and assessment of the evidence base on outcome measurement for pre-hospital care and a consensus studies to identify measures relevant to patients and NHS staff. 2: Linking pre-hospital data with other patient data sources – creating a single dataset that links ambulance service electronic care records with routinely collected Hospital Episode Statistics (HES) and national mortality data. 3. Development of risk adjustment models for outcomes in patients attended by the ambulance service – using the linked data to develop risk adjustment tools that will allow patient differences to be taken into account and differences between expected and actual outcomes to be detected. Particular emphasis will be made to include the broad EMS population and not specific conditions as has been the case in the past. 4. Testing the risk adjustment models to assess if they can be used to measure effectiveness and quality – exploring the practical application of the measures by using them to assess if different ways of providing ambulance service care result in different consequences for patients. Outputs, outcomes and impact The programme will: • Provide a summary of relevant evidence on pre-hospital care outcome measurement • Develop a method for linking healthcare information into a format that can be used to support quality improvement, is acceptable to patients and complies with information legislation • Develop population based models for measuring the impact of pre-hospital care that can be used to monitor quality and safety, evaluate new service innovations and support quality improvement • Provide added value by using routine information and NHS infrastructure to operationalise the process and outcome models so that they will be of use across the NHS Progress to date The programme commenced in June 2011 and ends in May 2016. Two systematic reviews of measures used to measure the impact of ambulance service care (one policy literature and one research literature based) have been completed as has a qualitative study of recent service users to identify aspects of service they value. Potential measures identified by these studies were presented at a consensus conference and then further refined in a Delphi study to prioritise and identify measures for further development. Linked data is currently being created and the next stage will be the development of risk adjusted predictive models for the final identified measures

Prehospital outcomes for ambulance service care: systematic review

Background: Ambulance service performance measurement has previously focused on response times and survival. We conducted a systematic review of the international literature on quality measures and outcomes relating to pre-hospital ambulance service care, aiming to identify a broad range of outcome measures to provide a more meaningful assessment of ambulance service care. Methods: We searched a number of electronic databases including CINAHL, the Cochrane Library, EMBASE, Medline, and Web of Science. For inclusion, studies had to report either research or evaluation conducted in a pre-hospital setting, published in the English language from 1982 to 2011, and reporting either outcome measures or specific outcome instruments. Results: Overall, 181 full-text articles were included: 83 (46%) studies from North America, 50 (28%) from Europe and 21 (12%) from the UK. A total of 176 articles were obtained after examining 257 full-text articles in detail from 5,088 abstracts screened. A further five papers were subsequently identified from references of the articles examined and studies known to the authors. There were 140 articles (77%) which contained at least one survival-related measure, 47 (34%) which included information about length of stay and 87 (48%) which identified at least one place of discharge as an outcome. Limitations: We encountered the problem of incomplete information, for instance studies not specifying which pain scales when these had been used or using survival without a specific time period. Conclusion and recommendations: In addition to measures relating to survival, length of stay and place of discharge, we identified 247 additional outcome measures. Few studies included patient reported or cost outcomes. By identifying a wide range of outcome measures this review will inform further research looking at the feasibility of using a wider range of outcome measures and developing new outcome measures in prehospital research and quality improvement

How should we measure ambulance service quality and performance?

The problem Ambulance services in England treat 6.5 million people per year but get no information about what happens to patients after discharge. This has led to a reliance on measuring response times rather than outcomes to assess how well services perform, and little opportunity for identifying problems and good practice or evaluating service developments. Research aim There is a lack of consensus on which outcome measures are important for pre-hospital care so we set out to address this within the Prehospital Outcomes for Evidence Based Evaluation (PhOEBE) research programme. Methods We conducted a two round Delphi study to prioritise outcome measures identified from a systematic review and a multi-stakeholder consensus event. 20 participants scored 57 measures over two rounds. Participants included policy makers and commissioners, clinical ambulance service and ambulance service operational groups. Outcomes were scored in three categories: patient outcomes; whole service measures and clinical management. Results Highly ranked patient outcome measures related to pain, survival, recontacts and patient experience. High ranking outcomes in the Clinical Management group related to compliance with protocols and guidelines and appropriateness and accuracy of triage. In the Whole Service measures group highly ranked measures related to completeness of clinical records, staff training and time to definitive care. Conclusions The next steps are to identify which measures are suitable for measuring with routine data; use a linked dataset to build predictive models and determine what aspects of care can predict good or poor outcomes (mortality and non-mortality); measure the effectiveness and quality of ambulance service care, and; assess the practical use of the measures and the linked data as a way to support quality improvement in the NHS

Taxonomy of usage problems for improving user-centric online health information provision

Consumer health information portals (HIP) are a popular means to provide quality health information via the Web. However complex usage problems in HIPs are still a major barrier to their success. A usage-driven approach, which places emphasis on improving online services based on learnings from the data of the interactions between users and the system, is crucial to ensuring sustainable and user-centred online health provision. Inspired by this idea, we present a taxonomy of usage problems that encompasses the dimensions of the content, the systems and users, focusing on a holistic understanding of usage problems. Our taxonomy is grounded on a literature analysis empirically validated through an analysis of usage-data captured from a consumer health information portal, operational for the past five years. By exploring how usage data highlights user problems, we also present strategies for health portal improvements based on better understandings of usage data. Benefits of usage-driven health portals in terms of smart learning capabilities to improve content and user satisfaction are discussed

Taxonomy of Usage Issues for Consumer-centric Online Health Information Provision

Consumers are increasingly using Internet portals when searching for relevant health information. Despite the broad range of health information portals (HIPs) available, usage problems with such portals are still widely recognized and reported. In this study, we analyzed usage data from an operational health information portal and identified ways in which these problems can be addressed. While previous usage data and log analysis research has focused more on user behaviors, query structures, and human-computer interaction issues, this study covers more comprehensive issues such as content. We describe a taxonomy of usage issues derived from a literature analysis. We describe how we validated and refined the taxonomy based on analyzing the usage data from an operational health portal. Findings from the usage data indicate that a range of content issues exist that lead to unsuccessful searches. The analysis also highlights that users’ ineffective information seeking strategies are not well supported by the system’s design. We use this taxonomy to propose a usage-driven, consumer-centered approach for dynamic improvements of HIPs. We also discuss the study’s limitations and directions for future research

Prioritising outcomes measures for ambulance service care: a three stage consensus study

Background Historically ambulance care quality and performance has been measured by response times rather than clinical need or effectiveness. This limits the evidence about the effectiveness of the care the ambulance service provides. The Pre-hospital Outcomes for Evidence Based Evaluation (PhOEBE) project is a 5 year research programme which aims to develop new ways of measuring the performance, quality and impact of ambulance service care that better reflect the care provided. Methods We held a 1 day consensus event with a multi-stakeholder group to identify and prioritise which features of ambulance service care are important and worth further development as measures of performance and quality. The day comprised a series of small group discussions to share opinions about measures identified from literature reviews and the opportunity for participants to add their ideas, together with a live vote to help identify and rank measures for further development. Time measures were considered in a separate online questionnaire. The results from the online questionnaire and Consensus Event were fed into a Delphi study. Results 43 participants attended the consensus event, representing a range of participant groups including patient and public, ambulance services, commissioners and policy groups. The top 5 outcome measures prioritised by the participants were: accuracy of dispatch decisions; completeness and accuracy of patient records; accuracy of call taker identification of different conditions or needs (e.g. heart attack, stroke, suitable for nurse advice); pain measurement and symptom relief; and patient experience. From the online survey of time measures the top 5 measures were time of call to time of arrival at scene; time symptoms start (e.g. chest pain) to time of treatment with balloon for heart attack; time taken to answer the emergency call; total time spent on scene; and incident or start of symptoms to calling the ambulance service. Conclusion Using consensus methods we have identified a set of outcome measures that can potentially be used to measure the performance and quality of ambulance service care. Some of these measures fit with existing government targets, such as response times and time taken to answer the emergency call. However, most measures are new ways of measuring ambulance service care and the importance of these measures and methods of measurement will be assessed as part of an on-going Delphi study

What outcome measures should be developed for pre-hospital care?: results of a consensus event

Background: The PhOEBE (Pre-hospital Outcomes for Evidence Based Evaluation) project is a 5-year research programme which aims to develop new ways of measuring the performance, quality and impact of ambulance service care. As part of this programme we conducted 2 systematic reviews to identify potential measures and held a consensus event to prioritise these measures. Methods: Actual or potential measures or indicators for assessing ambulance service performance or quality of care were identified from two systematic reviews and categorised as clinical management, operational or patient based measures. Time measures were considered separately. We held a consensus event with participants representing clinicians, ambulance operations, commissioners, policy and academic research. Three small group discussion sessions were held and after each session we used turning point software for participants to electronically vote whether they thought each potential measure was essential, desirable or irrelevant. Results: 42 participants took part and discussed and voted on 52 different measures. They could also add measures. The top 5 ranked operational measures were concerned with completeness and accuracy of records; accuracy of triage; appropriateness of service; ambulance training and ambulance utilisation. Top 5 clinical measures were accuracy of dispatch decisions, accuracy of problem identification, compliance with end of life care plans; patient safety and compliance with protocols. The top 5 patient measures were measurement and relief of pain; patient experience; return of spontaneous circulation; complications from care and survival. Conclusions: Accuracy of different types of decision making and compliance with management protocols predominated as essential with pain management the most important patient measure. Management of end of life care was identified by participants. The electronic voting system which provided instant real time feedback was well received by participants. The next stage is a Delphi survey to further refine these measures and include time interval measures

Systematic review of pre-hospital outcomes for evidence-based evaluation of ambulance service care

Background: Ambulance service performance measurement has previously focused on response times and survival (particularly from out-of-hospital cardiac arrest). The PhOEBE (Pre-hospital Outcomes for Evidence Based Evaluation) project is a 5-year research programme which aims to develop new ways of measuring the performance, quality and impact of ambulance service care. As part of this programme we conducted a systematic review of the international literature on quality measures and outcomes relating to pre-hospital ambulance service care, aiming to identify a broad range of outcome measures to provide a more meaningful assessment of ambulance service care. Methods: We searched a number of electronic databases including CINAHL, the Cochrane Library, EMBASE, Medline, and Web of Science. For inclusion, studies had to report either research or evaluation conducted in a pre-hospital setting and published in the English language from 1982 to 2011, reporting either outcome measures or specific outcome instruments. Results: Overall, 181 full-text articles were included: 83 (46%) studies from North America, 50 (28%) from Europe and 21 (12%) from the United Kingdom. Initially, 176 articles were included after examining 257 full-text articles from 5,088 abstracts screened. A further five papers were subsequently identified from references of the articles examined and studies known to the authors. There were 140 articles (77%) which contained at least one survival-related measure, while 47 (26%) included information about length of stay and 87 (48%) identified at least one place of discharge as an outcome. Conclusion: In addition to measures relating to survival, length of stay and place of discharge, we identified over 100 additional outcome measures. Few studies included patient reported outcomes or economic outcomes. By identifying a range of outcome measures, this review will inform the use of a greater range of outcome measures and development of new outcome measures in pre-hospital research and quality improvement

Prioritising outcomes measures for ambulance service care: a three stage consensus study

Background Historically ambulance care quality and performance has been measured by response times rather than clinical need or effectiveness. This limits the evidence about the effectiveness of the care the ambulance service provides. The Pre-hospital Outcomes for Evidence Based Evaluation (PhOEBE) project is a 5 year research programme which aims to develop new ways of measuring the performance, quality and impact of ambulance service care that better reflect the care provided. Methods We held a 1 day consensus event with a multi-stakeholder group to identify and prioritise which features of ambulance service care are important and worth further development as measures of performance and quality. The day comprised a series of small group discussions to share opinions about measures identified from literature reviews and the opportunity for participants to add their ideas, together with a live vote to help identify and rank measures for further development. Time measures were considered in a separate online questionnaire. The results from the online questionnaire and Consensus Event were fed into a Delphi study. Results 43 participants attended the consensus event, representing a range of participant groups including patient and public, ambulance services, commissioners and policy groups. The top 5 outcome measures prioritised by the participants were: Accuracy of dispatch decisions; Completeness and accuracy of patient records; Accuracy of call taker identification of different conditions or needs (e.g. heart attack, stroke, suitable for nurse advice); Pain measurement and symptom relief; Patient experience. From the online survey of time measures the top 5 measures were Time of call to time of arrival at scene; Time symptoms start (e.g. chest pain) to time of treatment with balloon for heart attack; Time taken to answer the emergency call; Total time spent on scene; Incident or start of symptoms to calling the ambulance service Conclusions Using consensus methods we have identified a set of outcome measures that can potentially be used to measure the performance and quality of ambulance service care. Some of these measures fit with existing government targets, such as response times and time taken to answer the emergency call. However most measures are new ways of measuring ambulance service care and the importance of these measures and methods of measurement will be assessed as part of an on-going Delphi study