Rapid evaluation of service innovations in health and social care: key considerations

Background: The Children and Young People’s Mental Health Trailblazer programme is funding the creation of new mental health support teams to work in schools and further education colleges. Mental health support teams directly support children and young people with ‘mild to moderate’ mental health problems and work with school and college staff to promote well-being for all. A new workforce of education mental health practitioners is being trained for the teams. / Objective(s): The National Institute for Health and Care Research Birmingham, RAND and Cambridge Evaluation Rapid Evaluation Centre and Policy Innovation and Evaluation Research Unit undertook an early evaluation of the Trailblazer programme to examine the development, implementation and early progress of mental health support teams in the programme’s first 25 ‘Trailblazer’ sites. / Design: A mixed-methods evaluation, comprising three work packages: 1. Establishing the baseline and understanding the development and early impacts of the Trailblazer sites, including two rounds of surveys with key informants and participating education settings in all 25 sites. 2. More detailed research in five purposively selected Trailblazer sites, including interviews with a range of stakeholders and focus groups with children and young people. 3. Scoping and developing options for a longer-term assessment of the programme’s outcomes and impacts. Fieldwork was undertaken between November 2020 and February 2022. The University of Birmingham Institute for Mental Health Youth Advisory Group was involved throughout the study, including co-producing the focus groups with children and young people. / Results: Substantial progress had been made implementing the programme, in challenging circumstances, and there was optimism about what it had the potential to achieve. The education mental health practitioner role had proven popular, but sites reported challenges in retaining education mental health practitioners, and turnover left mental health support teams short-staffed and needing to re-recruit. Education settings welcomed additional mental health support and reported positive early outcomes, including staff feeling more confident and having faster access to advice about mental health issues. At the same time, there were concerns about children who had mental health problems that were more serious than ‘mild to moderate’ but not serious enough to be accepted for specialist help, and that the interventions offered were not working well for some young people. Mental health support teams were generally spending more time supporting children with mental health problems than working with education settings to develop ‘whole school’ approaches to mental health and well-being, and service models in some sites appeared to be more clinically oriented, with a strong focus on mental health support teams’ therapeutic functions. / Limitations: Despite efforts to maximise participation, survey response rates were relatively low and some groups were less well represented than others. We were not able to gather sufficiently detailed data to develop a typology of Trailblazer sites, as was planned. / Conclusions: Key lessons for future programme implementation include: – Whether mental health support teams should expand support to children and young people with more complex and serious mental health problems. – How to keep the twin aims of prevention and early intervention in balance. – How to retain education mental health practitioners once trained. / Future work: The findings have important implications for the design of a longer-term impact evaluation of the programme, which is due to commence in summer 2023. / Study registration: Ethical approval from the University of Birmingham (ERN_19-1400 – RG_19-190) and London School of Hygiene and Tropical Medicine (Ref: 18040) and Health Research Authority approval (IRAS 270760). / Funding: The Birmingham, RAND and Cambridge Evaluation Rapid Evaluation Centre is funded by the National Institute for Health and Care Research Health Services and Delivery Research programme (HSDR 16/138/31). The Policy Innovation and Evaluation Research Unit is funded by the NIHR Policy Research Programme (PR-PRU-1217-20602)

The implementation of remote home monitoring models during the COVID-19 pandemic in England.

BACKGROUND: There is a paucity of evidence for the implementation of remote home monitoring for COVID-19 infection. The aims of this study were to identify the key characteristics of remote home monitoring models for COVID-19 infection, explore the experiences of staff implementing these models, understand the use of data for monitoring progress against outcomes, and document variability in staffing and resource allocation. METHODS: This was a multi-site mixed methods study conducted between July and August 2020 that combined qualitative and quantitative approaches to analyse the implementation and impact of remote home monitoring models developed during the first wave of the COVID-19 pandemic in England. The study combined interviews (n = 22) with staff delivering these models across eight sites in England with the collection and analysis of data on staffing models and resource allocation. FINDINGS: The models varied in relation to the healthcare settings and mechanisms used for patient triage, monitoring and escalation. Implementation was embedded in existing staff workloads and budgets. Good communication within clinical teams, culturally-appropriate information for patients/carers and the combination of multiple approaches for patient monitoring (app and paper-based) were considered facilitators in implementation. The mean cost per monitored patient varied from £400 to £553, depending on the model. INTERPRETATION: It is necessary to provide the means for evaluating the effectiveness of these models, for example, by establishing comparator data. Future research should also focus on the sustainability of the models and patient experience (considering the extent to which some of the models exacerbate existing inequalities in access to care)

Estimating the long-term impact of a prophylactic human papillomavirus 16/18 vaccine on the burden of cervical cancer in the UK

To predict the public health impact on cervical disease by introducing human papillomavirus (HPV) vaccination in the United Kingdom, we developed a mathematical model that can be used to reflect the impact of vaccination in different countries with existing screening programmes. Its use is discussed in the context of the United Kingdom. The model was calibrated with published data. The impact of vaccination on cervical cancer and deaths, precancerous lesions and screening outcomes were estimated for a vaccinated cohort of 12-year-old girls, among which it is estimated that there would be a reduction of 66% in the prevalence of high-grade precancerous lesions and a 76% reduction in cervical cancer deaths. Estimates for various other measures of the population effects of vaccination are also presented. We concluded that it is feasible to forecast the potential effects of HPV vaccination in the context of an existing national screening programme. Results suggest a sizable reduction in the incidence of cervical cancer and related deaths. Areas for future research include investigation of the beneficial effects of HPV vaccination on infection transmission and epidemic dynamics, as well as HPV-related neoplasms in other sites

Rapid Evaluation of the Special Measures for Quality and Challenged Provider Regimes: A Mixed-Methods Study

Background: Healthcare organisations in England rated as inadequate for leadership and one other domain enter Special Measures for Quality (SMQ) to receive support and oversight. A ‘watch list’ of challenged providers (CPs) at risk of entering SMQ also receive support. Knowledge is limited about whether the support interventions drive improvements in quality, their costs, and whether they strike the right balance between support and scrutiny. Objective: Analyse trust responses to the implementation of a) interventions for SMQ trusts and b) interventions for CP trusts to determine their impact on these organisations' capacity to achieve and sustain quality improvements. Design: Rapid research comprising five inter-related workstreams: 1. Literature review using systematic methods. 2. Analysis of policy documents and interviews at national level. 3. Eight multi-site, mixed method trust case studies. 4. Analysis of national performance and workforce indicators. 5. Economic analysis. Results: SMQ/CP were intended to be “support” programmes. SMQ/CP had an emotional impact on staff. Perceptions of NHSI interventions were mixed overall. Senior leadership teams were a key driver of change, with strong clinical input vital. Local systems have a role in improvement. Trusts focus efforts to improve across multiple domains. Internal and external factors contribute to positive performance trajectories. Nationally, only 15.8% of SMQ trusts exited within 24 months. Relative to national trends, entry into SMQ/CP corresponded to positive changes in 4-hour waits in Emergency Departments, mortality and delayed transfers of care. Trends in staff sickness and absence improved after trusts left SMQ/CP. There was some evidence that staff survey results improve. No association was found between SMQ/CP and referral to treatment times or cancer waiting times. The largest components of NHSI spending in case studies were interventions directed at 'training on cultural change' (33.6%), 'workforce quality and safety' (21.7%) and 'governance and assurance' (18.4%). Impact of SMQ on financial stability was equivocal; most trusts exiting SMQ experienced the same financial stability before and after exiting. Limitations: The rapid research design and one-year timeframe precludes longitudinal observations of trusts and local systems. The small number of indicators limited the quantitative analysis of impact. Measuring workforce effects was limited by data availability. Conclusions: Empirical evidence of positive impacts from SMQ/CP were identified, however, perceptions were mixed. Key lessons: • Time is needed to implement and embed changes. • Ways to mitigate emotional costs and stigma are needed. • Support strategies should be more trust specific. • Poor organisational performance needs to be addressed within local systems. • Senior leadership teams with stability, strong clinical input and previous SMQ experience helped enact change. • Organisation-wide quality improvement strategies and capabilities are needed. • Staff engagement and an open listening culture promote continuous learning and a quality improvement ‘mindset’, critical for sustainable improvement. • Need to consider level of sustainable funds required to improve patients’ outcomes. Future work: Evaluating recent changes to the regimes; role of local systems; longitudinal approaches. Study registration: Review protocol registered with PROSPERO (CRD: 42019131024). Funding: The National Institute for Health Research Health Services and Delivery Research programme (16/138/17 – Rapid Service Evaluation Research Team)

Patient and staff experiences of using technology-enabled and analogue models of remote home monitoring for COVID-19 in England: A mixed-method evaluation

Objective: To evaluate patient and staff experiences of using technology-enabled (‘tech-enabled’) and analogue remote home monitoring models for COVID-19, implemented in England during the pandemic. // Methods: Twenty-eight sites were selected for diversity in a range of criteria (e.g. pre-hospital or early discharge service, mode of patient data submission). Between February and May 2021, we conducted quantitative surveys with patients, carers and staff delivering the service, and interviewed patients, carers, and staff from 17 of the 28 services. Quantitative data were analysed using descriptive statistics and both univariate and multivariate analyses. Qualitative data were interpreted using thematic analysis. // Results: Twenty-one sites adopted mixed models whereby patients could submit their symptoms using either tech-enabled (app, weblink, or automated phone calls) or analogue (phone calls with a health professional) options; seven sites offered analogue-only data submission (phone calls or face-to-face visits with a health professional). Sixty-two patients and carers were interviewed, and 1069 survey responses were received (18 % response rate). Fifty-eight staff were interviewed, and 292 survey responses were received (39 % response rate). Patients who used tech-enabled modes tended to be younger (p = 0.005), have a higher level of education (p = 0.011), and more likely to identify as White British (p = 0.043). Most patients found relaying symptoms easy, regardless of modality, though many received assistance from family or friends. Staff considered the adoption of mixed delivery models beneficial, enabling them to manage large patient numbers and contact patients for further assessment as needed; however, they suggested improvements to the functionality of systems to better fit clinical and operational needs. Human contact was important in all remote home monitoring options. // Conclusions: Organisations implementing tech-enabled remote home monitoring at scale should consider adopting mixed models which can accommodate patients with different needs; focus on the usability and interoperability of tech-enabled platforms; and encourage digital inclusivity for patients

Screening for cervical Cancer in high-risk populations: DNA pap test or hybrid capture II test alone?

This study was designed to evaluate whether Hybrid Capture II (HC2) test alone refer women to colposcopy as appropriately as DNA Papanicolaou (Pap) test, in the context of a high-risk group of women using the recently validated DNACitoliq LBC system. Women with suspected cervical disease were included in this crosssectional study at a tertiary center in São Paulo, Brazil, for further workup. All women had cervical material collected for LBC and HC2 for high-risk human papillomavirus (hrHPV)-DNA test. Irrespective of cytology and HC2 results, colposcopy, and cervical biopsy when applicable, was systematically performed. All tests were performed blindly. Sensitivity, specificity, positive and negative predictive values, and overall accuracy of both methods were computed in relation to histology. A total of 1,080 women were included: 36.4% (393/1080) had ACUS+, 10.2% (110/1080) were high-grade squamous intraepithelial lesions (HSIL) or cancer. Mean age was 33.5 years. All women underwent colposcopy, and cervical biopsies were performed in 38.4% (415/1080): 33% (137/415) of the biopsies were negative, 14.4% (155/415) were low-grade squamous intraepithelial lesions (LSIL), 10.7% (116/415) were HSIL, and 0.6% (7/415) were cancer. HC2 sensitivity to diagnose biopsy-proven HSIL was 100%. Because all HSIL cases had a positive HC2 test, sensitivity could not be improved by adding LBC. Specificity and positive and negative predictive values of DNA Pap were not significantly different from HC2 test alone when considering LSIL+ histology as ‘‘gold standard’’ and HSIL+ histology. As a screening strategy for women with high-risk for cervical cancer, DNA Pap test does not seem to add substantially to HC2 alone in terms of appropriately referring to colposcopy

Monitoring clinical quality in rare disease services – experience in England

After some well-publicised problems with paediatric cardiac surgery, there has been great interest in England in monitoring clinical quality in specialised medical services. The National Commissioning Group plans, funds and monitors a set of highly specialised services for the National Health Service in England. We have developed systems for monitoring clinical quality that perform two interrelated but distinct functions: performance measurement and performance improvement. The aim is to collect information on all patients seen during each year – a 100% consecutive case series. Generally, there is no conceptual difficulty identifying an appropriate outcome for surgical interventions: the indication for surgery usually defines the outcome to monitor. This is not so for the medical and psychiatric services, where the relevant outcome to monitor is sometimes not obvious. There are a number of problems in interpreting, and acting on, outcome data for rare conditions and treatments. These problems include statistical problems due to small numbers, the need to risk adjust data and coding problems

Rapid evaluation of the Special Measures for Quality and challenged provider regimes: a mixed-methods study

Background: Health-care organisations in England that are rated as inadequate for leadership and one other domain enter the Special Measures for Quality regime to receive support and oversight. A ‘watch list’ of challenged providers that are at risk of entering Special Measures for Quality also receive support. Knowledge is limited about whether or not the support interventions drive improvements in quality, the costs of the support interventions and whether or not the support interventions strike the right balance between support and scrutiny. Objective: To analyse the responses of trusts to the implementation of (1) interventions for Special Measures for Quality trusts and (2) interventions for challenged provider trusts to determine their impact on these organisations’ capacity to achieve and sustain quality improvements. Design: This was rapid research comprising five interrelated workstreams: (1) a literature review using systematic methods; (2) an analysis of policy documents and interviews at the national level; (3) eight multisite, mixed-methods trust case studies; (4) an analysis of national performance and workforce indicators; and (5) an economic analysis. Results: The Special Measures for Quality/challenged provider regimes were intended to be ‘support’ programmes. Special Measures for Quality/challenged provider regimes had an emotional impact on staff. Perceptions of NHS Improvement interventions were mixed overall. Senior leadership teams were a key driver of change, with strong clinical input being vital. Local systems have a role in improvement. Trusts focus efforts to improve across multiple domains. Internal and external factors contribute to positive performance trajectories. Nationally, only 15.8% of Special Measures for Quality trusts exited the regime in 24 months. Entry into Special Measures for Quality/challenged provider regimes resulted in changes in quality indicators (such the number of patients waiting in emergency departments for more than 4 hours, mortality and the number of delayed transfers of care) that were more positive than national trends. The trends in staff sickness and absence improved after trusts left Special Measures for Quality/challenged provider regimes. There was some evidence that staff survey results improved. No association was found between Special Measures for Quality/challenged provider regimes and referral to treatment times or cancer treatment waiting times. NHS Improvement spending in case study trusts was mostly directed at interventions addressing ‘training on cultural change’ (33.6%), ‘workforce quality and safety’ (21.7%) and ‘governance and assurance’ (18.4%). The impact of Special Measures for Quality on financial stability was equivocal; most trusts exiting Special Measures for Quality experienced the same financial stability before and after exiting. Limitations: The rapid research design and 1-year time frame precludes longitudinal observations of trusts and local systems. The small number of indicators limited the quantitative analysis of impact. Measurement of workforce effects was limited by data availability. Conclusions: Empirical evidence of positive impacts of Special Measures for Quality/challenged provider regimes were identified; however, perceptions were mixed. Key lessons were that (1) time is needed to implement and embed changes; (2) ways to mitigate emotional costs and stigma are needed; (3) support strategies should be more trust specific; (4) poor organisational performance needs to be addressed within local systems; (5) senior leadership teams with stability, strong clinical input and previous Special Measures for Quality experience helped to enact change; (6) organisation-wide quality improvement strategies and capabilities are needed; (7) staff engagement and an open-listening culture promote continuous learning and a quality improvement ‘mindset’, which is critical for sustainable improvement; and (8) consideration of the level of sustainable funds required to improve patients’ outcomes is needed. Future work: Future work could include evaluating recent changes to the regimes, the role of local systems and longitudinal approaches. Study registration: The review protocol is registered with PROSPERO (CRD42019131024). Funding: This project was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme and will be published in full in Health and Social Care Delivery Research; Vol. 11, No. 19. See the NIHR Journals Library website for further project information