Practical guidance on undertaking a service evaluation

This article describes the basic principles of evaluation, focusing on the evaluation of healthcare services. It emphasises the importance of evaluation in the current healthcare environment and the requirement for nurses to understand the essential principles of evaluation. Evaluation is defined in contrast to audit and research, and the main theoretical approaches to evaluation are outlined, providing insights into the different types of evaluation that may be undertaken. The essential features of preparing for an evaluation are considered, and guidance provided on working ethically in the NHS. It is important to involve patients and the public in evaluation activity, offering essential guidance and principles of best practice. The authors discuss the main challenges of undertaking evaluations and offer recommendations to address these, drawing on their experience as evaluators

Effectiveness of a symptom-clinic intervention delivered by general practitioners with an extended role for people with multiple and persistent physical symptoms in England:the Multiple Symptoms Study 3 pragmatic, multicentre, parallel-group, individually randomised controlled trial

BACKGROUND: People with multiple and persistent physical symptoms have impaired quality of life and poor experiences of health care. We aimed to evaluate the effectiveness of a community-based symptom-clinic intervention in people with multiple and persistent physical symptoms, hypothesising that this symptoms clinic plus usual care would be superior to usual care only.METHODS: The Multiple Symptoms Study 3 was a pragmatic, multicentre, parallel-group, individually randomised controlled trial conducted in 108 general practices in the UK National Health Service in four regions of England between Dec 6, 2018, and June 30, 2023. Participants were individually randomised (1:1) to the symptom-clinic intervention plus usual care or to usual care only via a computer-generated, pseudo-random list stratified by trial centre. Allocation was done by the trial statistician and concealed with a centralised, web-based randomisation system; masking participants was not possible due to the nature of the intervention. The symptom-clinic intervention was a sequence of up to four medical consultations that aimed to elicit a detailed clinical history, fully hear and validate the participant, offer rational explanations for symptoms, and assist the participant to develop ways of managing their symptoms; it was delivered by general practitioners with an extended role. The primary outcome was Patient Health Questionnaire-15 (PHQ-15) score 52 weeks after randomisation, analysed by intention to treat. The trial is registered on the ISRCTN registry (ISRCTN57050216).FINDINGS: 354 participants were randomly assigned; 178 (50%) were assigned to receive the community-based symptoms clinic plus usual care and 176 (50%) were assigned to receive usual care only. At the primary-outcome point of 52 weeks, PHQ-15 scores were 14·1 (SD 3·7) in the group receiving usual care and 12·2 (4·5) in the group receiving the intervention. The adjusted between-group difference of -1·82 (95% CI -2·67 to -0·97) was statistically significantly in favour of the intervention group (p&lt;0·0001). There were 39 adverse events in the group receiving usual care and 36 adverse events in the group receiving the intervention. There were no statistically significant between-group differences in the proportion of participants who had non-serious adverse events (-0·03, 95% CI -0·11 to 0·05) or serious adverse events (0·02, -0·02 to 0·07). No serious adverse event was deemed to be related to the trial intervention.INTERPRETATION: Our symptom-clinic intervention, which focused on explaining persistent symptoms to participants in order to support self-management, led to sustained improvement in multiple and persistent physical symptoms.FUNDING: UK National Institute for Health and Care Research.</p

Supporting physical activity through co-production in people with severe mental ill health (SPACES): protocol for a randomised controlled feasibility trial.

Background Severe mental ill health (SMI) includes schizophrenia, bipolar disorder and schizoaffective disorder and is associated with premature deaths when compared to people without SMI. Over 70% of those deaths are attributed to preventable health conditions, which have the potential to be positively affected by the adoption of healthy behaviours, such as physical activity. People with SMI are generally less active than those without and face unique barriers to being physically active. Physical activity interventions for those with SMI demonstrate promise, however, there are important questions remaining about the potential feasibility and acceptability of a physical activity intervention embedded within existing NHS pathways. Method This is a two-arm multi-site randomised controlled feasibility trial, assessing the feasibility and acceptability of a co-produced physical activity intervention for a full-scale trial across geographically dispersed NHS mental health trusts in England. Participants will be randomly allocated via block, 1:1 randomisation, into either the intervention arm or the usual care arm. The usual care arm will continue to receive usual care throughout the trial, whilst the intervention arm will receive usual care plus the offer of a weekly, 18-week, physical activity intervention comprising walking and indoor activity sessions and community taster sessions. Another main component of the intervention includes one-to-one support. The primary outcome is to investigate the feasibility and acceptability of the intervention and to scale it up to a full-scale trial, using a short proforma provided to all intervention participants at follow-up, qualitative interviews with approximately 15 intervention participants and 5 interventions delivery staff, and data on intervention uptake, attendance, and attrition. Usual care data will also include recruitment and follow-up retention. Secondary outcome measures include physical activity and sedentary behaviours, body mass index, depression, anxiety, health-related quality of life, healthcare resource use, and adverse events. Outcome measures will be taken at baseline, three, and six-months post randomisation. Discussion This study will determine if the physical activity intervention is feasible and acceptable to both participants receiving the intervention and NHS staff who deliver it. Results will inform the design of a larger randomised controlled trial assessing the clinical and cost effectiveness of the intervention. Trial registration ISRCTN: ISRCTN83877229. Registered on 09.09.2022

The Journeying through Dementia psychosocial intervention versus usual care study: a two-arm, phase 3, superiority randomised controlled trial

Background There is an urgent clinical need for evidence based psychosocial interventions for people with mild dementia. We aimed to determine clinical and cost-effectiveness of Journeying through Dementia (JtD), an intervention designed to promote wellbeing and independence in people with mild dementia. Methods We conducted a phase 3, two-arm parallel group, superiority, individually randomised controlled trial. Thirteen NHS sites across England recruited community-dwelling people with mild dementia and Mini-Mental State Examination score of ≥18. Centralised randomisation (1:1) was conducted for participants to receive JtD plus usual care or usual care (control), stratified by site. The JtD Intervention comprised 12 group and four one-to-one sessions, delivered in the community at each site. The primary endpoint was Dementia Related Quality of Life (DEMQOL), eight-months postrandomisation. Only outcome assessors were blinded. Analysis was by intention-to-treat. Costeffectiveness analysis reported cost per quality adjusted life year (QALY) from a UK NHS and social care perspective. Findings In total 480 people were randomised to intervention (241) or control (239) arms. Intervention adherence was very good. Mean DEMQOL scores at eight months were intervention arm 93·3 (SD 13·0, n=191) and control 91·9 (SD 14·6, n=197). Difference in means was 0·9 (95% CI -1·2 to 3·0) after adjustment for covariates, lower than that identified as clinically meaningful. Incremental cost per QALY ranged from £88K to -£205K, indicating that JtD was not cost-effective. Unrelated serious adverse events were experienced by 17% (40/241) of intervention and 15% (35/239) of control participants. Interpretation In common with other recently reported studies the JtD intervention was not provenly effective. However, this complex trial successfully recruited and retained people with dementia without necessarily involving carers, and people with dementia they were actively involved as participants and study advisors throughout. Further research into methods of measuring small, meaningful changes in this population is needed. Questions remain regarding how services can match the complex, diverse and individual needs of people with mild dementia, and how interventions to meet such needs can be delivered at scale

Optimal pharmacotherapy pathway in adults with diabetic peripheral neuropathic pain: the OPTION-DM RCT

Background: The mainstay of treatment for diabetic peripheral neuropathic pain is pharmacotherapy, but the current National Institute for Health and Care Excellence guideline is not based on robust evidence, as the treatments and their combinations have not been directly compared. Objectives: To determine the most clinically beneficial, cost-effective and tolerated treatment pathway for diabetic peripheral neuropathic pain. Design: A randomised crossover trial with health economic analysis. Setting: Twenty-one secondary care centres in the UK. Participants: Adults with diabetic peripheral neuropathic pain with a 7-day average self-rated pain score of ≥ 4 points (Numeric Rating Scale 0–10). Interventions: Participants were randomised to three commonly used treatment pathways: (1) amitriptyline supplemented with pregabalin, (2) duloxetine supplemented with pregabalin and (3) pregabalin supplemented with amitriptyline. Participants and research teams were blinded to treatment allocation, using overencapsulated capsules and matching placebos. Site pharmacists were unblinded. Outcomes: The primary outcome was the difference in 7-day average 24-hour Numeric Rating Scale score between pathways, measured during the final week of each pathway. Secondary end points included 7-day average daily Numeric Rating Scale pain score at week 6 between monotherapies, quality of life (Short Form questionnaire-36 items), Hospital Anxiety and Depression Scale score, the proportion of patients achieving 30% and 50% pain reduction, Brief Pain Inventory – Modified Short Form items scores, Insomnia Severity Index score, Neuropathic Pain Symptom Inventory score, tolerability (scale 0–10), Patient Global Impression of Change score at week 16 and patients’ preferred treatment pathway at week 50. Adverse events and serious adverse events were recorded. A withintrial cost–utility analysis was carried out to compare treatment pathways using incremental costs per quality-adjusted life-years from an NHS and social care perspective. Results: A total of 140 participants were randomised from 13 UK centres, 130 of whom were included in the analyses. Pain score at week 16 was similar between the arms, with a mean difference of –0.1 points (98.3% confidence interval –0.5 to 0.3 points) for duloxetine supplemented with pregabalin compared with amitriptyline supplemented with pregabalin, a mean difference of –0.1 points (98.3% confidence interval –0.5 to 0.3 points) for pregabalin supplemented with amitriptyline compared with amitriptyline supplemented with pregabalin and a mean difference of 0.0 points (98.3% confidence interval –0.4 to 0.4 points) for pregabalin supplemented with amitriptyline compared with duloxetine supplemented with pregabalin. Results for tolerability, discontinuation and quality of life were similar. The adverse events were predictable for each drug. Combination therapy (weeks 6–16) was associated with a further reduction in Numeric Rating Scale pain score (mean 1.0 points, 98.3% confidence interval 0.6 to 1.3 points) compared with those who remained on monotherapy (mean 0.2 points, 98.3% confidence interval –0.1 to 0.5 points). The pregabalin supplemented with amitriptyline pathway had the fewest monotherapy discontinuations due to treatment-emergent adverse events and was most commonly preferred (most commonly preferred by participants: amitriptyline supplemented with pregabalin, 24%; duloxetine supplemented with pregabalin, 33%; pregabalin supplemented with amitriptyline, 43%; p = 0.26). No single pathway was superior in cost-effectiveness. The incremental gains in quality-adjusted life-years were small for each pathway comparison [amitriptyline supplemented with pregabalin compared with duloxetine supplemented with pregabalin –0.002 (95% confidence interval –0.011 to 0.007) quality-adjusted life-years, amitriptyline supplemented with pregabalin compared with pregabalin supplemented with amitriptyline –0.006 (95% confidence interval –0.002 to 0.014) qualityadjusted life-years and duloxetine supplemented with pregabalin compared with pregabalin supplemented with amitriptyline 0.007 (95% confidence interval 0.0002 to 0.015) quality-adjusted life-years] and incremental costs over 16 weeks were similar [amitriptyline supplemented with pregabalin compared with duloxetine supplemented with pregabalin −£113 (95% confidence interval −£381 to £90), ABSTRACT NIHR Journals Library www.journalslibrary.nihr.ac.uk viii amitriptyline supplemented with pregabalin compared with pregabalin supplemented with amitriptyline £155 (95% confidence interval −£37 to £625) and duloxetine supplemented with pregabalin compared with pregabalin supplemented with amitriptyline £141 (95% confidence interval −£13 to £398)]. Limitations: Although there was no placebo arm, there is strong evidence for the use of each study medication from randomised placebo-controlled trials. The addition of a placebo arm would have increased the duration of this already long and demanding trial and it was not felt to be ethically justifiable. Future work: Future research should explore (1) variations in diabetic peripheral neuropathic pain management at the practice level, (2) how OPTION-DM (Optimal Pathway for TreatIng neurOpathic paiN in Diabetes Mellitus) trial findings can be best implemented, (3) why some patients respond to a particular drug and others do not and (4) what options there are for further treatments for those patients on combination treatment with inadequate pain relief. Conclusions: The three treatment pathways appear to give comparable patient outcomes at similar costs, suggesting that the optimal treatment may depend on patients’ preference in terms of side effects. Trial registration: The trial is registered as ISRCTN17545443 and EudraCT 2016-003146-89. Funding: This project was funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment programme, and will be published in full in Health Technology Assessment; Vol. 26, No. 39. See the NIHR Journals Library website for further project information

Safety and efficacy of autologous haematopoietic stem-cell transplantation with low-dose cyclophosphamide mobilisation and reduced intensity conditioning versus standard of care in refractory Crohn's disease (ASTIClite): an open-label, multicentre, randomised controlled trial

Background: A previous controlled trial of autologous haematopoietic stem-cell transplantation (HSCT) in patients with refractory Crohn's disease did not meet its primary endpoint and reported high toxicity. We aimed to assess the safety and efficacy of HSCT with an immune-ablative regimen of reduced intensity versus standard of care in this patient population. Methods: This open-label, multicentre, randomised controlled trial was conducted in nine National Health Service hospital trusts across the UK. Adults (aged 18–60 years) with active Crohn's disease on endoscopy (Simplified Endoscopic Score for Crohn's Disease [SES-CD] ulcer sub-score of ≥2) refractory to two or more classes of biological therapy, with no perianal or intra-abdominal sepsis or clinically significant comorbidity, were recruited. Participants were centrally randomly assigned (2:1) to either HSCT with a reduced dose of cyclophosphamide (intervention group) or standard care (control group). Randomisation was stratified by trial site by use of random permuted blocks of size 3 and 6. Patients in the intervention group underwent stem-cell mobilisation (cyclophosphamide 1 g/m2 with granulocyte colony-stimulating factor (G-CSF) 5 μg/kg) and stem-cell harvest (minimum 2·0 × 106 CD34+ cells per kg), before conditioning (fludarabine 125 mg/m2, cyclophosphamide 120 mg/kg, and rabbit anti-thymocyte globulin [thymoglobulin] 7·5 mg/kg in total) and subsequent stem-cell reinfusion supported by G-CSF. Patients in the control group continued any available conventional, biological, or nutritional therapy. The primary outcome was absence of endoscopic ulceration (SES-CD ulcer sub-score of 0) without surgery or death at week 48, analysed in the intention-to-treat population by central reading. This trial is registered with the ISRCTN registry, 17160440. Findings: Between Oct 18, 2018, and Nov 8, 2019, 49 patients were screened for eligibility, of whom 23 (47%) were randomly assigned: 13 (57%) to the intervention group and ten (43%) to the control group. In the intervention group, ten (77%) participants underwent HSCT and nine (69%) reached 48-week follow-up; in the control group, nine (90%) reached 48-week follow-up. The trial was halted in response to nine reported suspected unexpected serious adverse reactions in six (46%) patients in the intervention group, including renal failure due to proven thrombotic microangiopathy in three participants and one death due to pulmonary veno-occlusive disease. At week 48, absence of endoscopic ulceration without surgery or death was reported in three (43%) of seven participants in the intervention group and in none of six participants in the control group with available data. Serious adverse events were more frequent in the intervention group (38 in 13 [100%] patients) than in the control group (16 in four [40%] patients). A second patient in the intervention group died after week 48 of respiratory and renal failure. Interpretation: Although HSCT with an immune-ablative regimen of reduced intensity decreased endoscopic disease activity, significant adverse events deem this regimen unsuitable for future clinical use in patients with refractory Crohn's disease. Funding: Efficacy and Mechanism Evaluation Programme, a Medical Research Council and National Institute for Health Research partnership

The impact of immediate breast reconstruction on the time to delivery of adjuvant therapy: the iBRA-2 study

Background: Immediate breast reconstruction (IBR) is routinely offered to improve quality-of-life for women requiring mastectomy, but there are concerns that more complex surgery may delay adjuvant oncological treatments and compromise long-term outcomes. High-quality evidence is lacking. The iBRA-2 study aimed to investigate the impact of IBR on time to adjuvant therapy. Methods: Consecutive women undergoing mastectomy ± IBR for breast cancer July–December, 2016 were included. Patient demographics, operative, oncological and complication data were collected. Time from last definitive cancer surgery to first adjuvant treatment for patients undergoing mastectomy ± IBR were compared and risk factors associated with delays explored. Results: A total of 2540 patients were recruited from 76 centres; 1008 (39.7%) underwent IBR (implant-only [n = 675, 26.6%]; pedicled flaps [n = 105,4.1%] and free-flaps [n = 228, 8.9%]). Complications requiring re-admission or re-operation were significantly more common in patients undergoing IBR than those receiving mastectomy. Adjuvant chemotherapy or radiotherapy was required by 1235 (48.6%) patients. No clinically significant differences were seen in time to adjuvant therapy between patient groups but major complications irrespective of surgery received were significantly associated with treatment delays. Conclusions: IBR does not result in clinically significant delays to adjuvant therapy, but post-operative complications are associated with treatment delays. Strategies to minimise complications, including careful patient selection, are required to improve outcomes for patients