Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation

Medical systematic reviews can be very costly and resource intensive. We explore how Large Language Models (LLMs) can support and be trained to perform literature screening when provided with a detailed set of selection criteria. Specifically, we instruction tune LLaMA and Guanaco models to perform abstract screening for medical systematic reviews. Our best model, Bio-SIEVE, outperforms both ChatGPT and trained traditional approaches, and generalises better across medical domains. However, there remains the challenge of adapting the model to safety-first scenarios. We also explore the impact of multi-task training with Bio-SIEVE-Multi, including tasks such as PICO extraction and exclusion reasoning, but find that it is unable to match single-task Bio-SIEVE's performance. We see Bio-SIEVE as an important step towards specialising LLMs for the biomedical systematic review process and explore its future developmental opportunities. We release our models, code and a list of DOIs to reconstruct our dataset for reproducibility

Gene expression profiling and expanded immunohistochemistry tests to guide selection of chemotherapy regimens in breast cancer management: a systematic review

OBJECTIVES: The aim of this report was to assess the clinical effectiveness of two Gene expression profiling (GEP) and two expanded immunohistochemistry (IHC) tests compared with current prognostic tools in guiding the use of adjuvant chemotherapy in patients with early breast cancer. METHODS: A systematic review of the evidence on clinical effectiveness of OncotypeDX, IHC4, MammaPrint and Mammostrat, compared with current clinical practice using clinicopathological parameters, in women with early breast cancer was conducted. Ten databases were searched to include citations to May 2016. RESULTS: Searches identified 7064 citations, of which 41 citations satisfied the criteria for the review. A narrative synthesis was performed. Evidence for OncotypeDX demonstrated the impact of the test on decision-making and there was some support for OncotypeDX predicting chemotherapy benefit. There were relatively lower levels of evidence for the other three tests included in the analysis. MammaPrint, Mammostrat and IHC4 tests were limited to a small number of studies. Limitations in relation to study design were identified for all tests. CONCLUSIONS: The evidence base for OncotypeDX is considered to be the most robust. Methodological weaknesses relating to heterogeneity of patient cohorts and issues arising from the retrospective nature of the evidence were identified. Further evidence is required for all of the tests using prospective randomised controlled trial data

EMA and NICE appraisal processes for cancer drugs : current status and uncertainties

The Evidence Review Group members that contributed to this editorial are funded by the UK NIHR HTA Programme. The views and opinions expressed are those of the authors and do not necessarily reflect those of the UK Department of Health and the NIHR.Peer reviewedPublisher PD

Cabazitaxel for Hormone-Relapsed Metastatic Prostate Cancer Previously Treated With a Docetaxel-Containing Regimen: An Evidence Review Group Perspective of a NICE Single Technology Appraisal.

As part of its single technology appraisal (STA) process, the National Institute for Health and Care Excellence (NICE) invited the company that manufactures cabazitaxel (Jevtana(®), Sanofi, UK) to submit evidence for the clinical and cost effectiveness of cabazitaxel for treatment of patients with metastatic hormone-relapsed prostate cancer (mHRPC) previously treated with a docetaxel-containing regimen. The School of Health and Related Research Technology Appraisal Group at the University of Sheffield was commissioned to act as the independent Evidence Review Group (ERG). The ERG produced a critical review of the evidence for the clinical and cost effectiveness of the technology based upon the company's submission to NICE. Clinical evidence for cabazitaxel was derived from a multinational randomised open-label phase III trial (TROPIC) of cabazitaxel plus prednisone or prednisolone compared with mitoxantrone plus prednisone or prednisolone, which was assumed to represent best supportive care. The NICE final scope identified a further three comparators: abiraterone in combination with prednisone or prednisolone; enzalutamide; and radium-223 dichloride for the subgroup of people with bone metastasis only (no visceral metastasis). The company did not consider radium-223 dichloride to be a relevant comparator. Neither abiraterone nor enzalutamide has been directly compared in a trial with cabazitaxel. Instead, clinical evidence was synthesised within a network meta-analysis (NMA). Results from TROPIC showed that cabazitaxel was associated with a statistically significant improvement in both overall survival and progression-free survival compared with mitoxantrone. Results from a random-effects NMA, as conducted by the company and updated by the ERG, indicated that there was no statistically significant difference between the three active treatments for both overall survival and progression-free survival. Utility data were not collected as part of the TROPIC trial, and were instead taken from the company's UK early access programme. Evidence on resource use came from the TROPIC trial, supplemented by both expert clinical opinion and a UK clinical audit. List prices were used for mitoxantrone, abiraterone and enzalutamide as directed by NICE, although commercial in-confidence patient-access schemes (PASs) are in place for abiraterone and enzalutamide. The confidential PAS was used for cabazitaxel. Sequential use of the advanced hormonal therapies (abiraterone and enzalutamide) does not usually occur in clinical practice in the UK. Hence, cabazitaxel could be used within two pathways of care: either when an advanced hormonal therapy was used pre-docetaxel, or when one was used post-docetaxel. The company believed that the former pathway was more likely to represent standard National Health Service (NHS) practice, and so their main comparison was between cabazitaxel and mitoxantrone, with effectiveness data from the TROPIC trial. Results of the company's updated cost-effectiveness analysis estimated a probabilistic incremental cost-effectiveness ratio (ICER) of £45,982 per quality-adjusted life-year (QALY) gained, which the committee considered to be the most plausible value for this comparison. Cabazitaxel was estimated to be both cheaper and more effective than abiraterone. Cabazitaxel was estimated to be cheaper but less effective than enzalutamide, resulting in an ICER of £212,038 per QALY gained for enzalutamide compared with cabazitaxel. The ERG noted that radium-223 is a valid comparator (for the indicated sub-group), and that it may be used in either of the two care pathways. Hence, its exclusion leads to uncertainty in the cost-effectiveness results. In addition, the company assumed that there would be no drug wastage when cabazitaxel was used, with cost-effectiveness results being sensitive to this assumption: modelling drug wastage increased the ICER comparing cabazitaxel with mitoxantrone to over £55,000 per QALY gained. The ERG updated the company's NMA and used a random effects model to perform a fully incremental analysis between cabazitaxel, abiraterone, enzalutamide and best supportive care using PASs for abiraterone and enzalutamide. Results showed that both cabazitaxel and abiraterone were extendedly dominated by the combination of best supportive care and enzalutamide. Preliminary guidance from the committee, which included wastage of cabazitaxel, did not recommend its use. In response, the company provided both a further discount to the confidential PAS for cabazitaxel and confirmation from NHS England that it is appropriate to supply and purchase cabazitaxel in pre-prepared intravenous-infusion bags, which would remove the cost of drug wastage. As a result, the committee recommended use of cabazitaxel as a treatment option in people with an Eastern Cooperative Oncology Group performance status of 0 or 1 whose disease had progressed during or after treatment with at least 225 mg/m(2) of docetaxel, as long as it was provided at the discount agreed in the PAS and purchased in either pre-prepared intravenous-infusion bags or in vials at a reduced price to reflect the average per-patient drug wastage

The use of rapid review methods in health technology assessments: 3 case studies.

BACKGROUND: Rapid reviews are of increasing importance within health technology assessment due to time and resource constraints. There are many rapid review methods available although there is little guidance as to the most suitable methods. We present three case studies employing differing methods to suit the evidence base for each review and outline some issues to consider when selecting an appropriate method. METHODS: Three recently completed systematic review short reports produced for the UK National Institute for Health Research were examined. Different approaches to rapid review methods were used in the three reports which were undertaken to inform the commissioning of services within the NHS and to inform future trial design. We describe the methods used, the reasoning behind the choice of methods and explore the strengths and weaknesses of each method. RESULTS: Rapid review methods were chosen to meet the needs of the review and each review had distinctly different challenges such as heterogeneity in terms of populations, interventions, comparators and outcome measures (PICO) and/or large numbers of relevant trials. All reviews included at least 10 randomised controlled trials (RCTs), each with numerous included outcomes. For the first case study (sexual health interventions), very diverse studies in terms of PICO were included. P-values and summary information only were presented due to substantial heterogeneity between studies and outcomes measured. For the second case study (premature ejaculation treatments), there were over 100 RCTs but also several existing systematic reviews. Data for meta-analyses were extracted directly from existing systematic reviews with new RCT data added where available. For the final case study (cannabis cessation therapies), studies included a wide range of interventions and considerable variation in study populations and outcomes. A brief summary of the key findings for each study was presented and narrative synthesis used to summarise results for each pair of interventions compared. CONCLUSIONS: Rapid review methods need to be chosen to meet both the nature of the evidence base of a review and the challenges presented by the included studies. Appropriate methods should be chosen after an assessment of the evidence base

Obinutuzumab with Bendamustine for Treating Follicular Lymphoma Refractory to Rituximab: An Evidence Review Group Perspective of a NICE Single Technology Appraisal

As part of its single technology appraisal process, the UK National Institute for Health and Care Excellence (NICE) invited the manufacturer of obinutuzumab (Roche) to submit evidence on its clinical and cost effectiveness when used in combination with bendamustine in patients with follicular lymphoma (FL) refractory to rituximab. The Evidence Review Group (ERG), the School of Health and Related Research Technology Appraisal Group at the University of Sheffield, produced a document summarising the key points from the company submission alongside a critical review. Efficacy for progression-free survival (PFS) and safety was positively demonstrated in the pivotal GADOLIN trial, which compared obinutuzumab in combination with bendamustine followed by obinutuzumab maintenance (O-Benda+O) against bendamustine monotherapy. Data on overall survival were immature. The company submitted a model-based economic analysis, including a patient access scheme. The ERG identified a number of limitations, in particular the absence of subgroup analysis and the approach used by the company to estimate overall survival (OS), which was more favourable to the intervention arm. The key uncertainty was the duration of the treatment effect on OS. This uncertainty is expected to be reduced when the final analysis of the GADOLIN trial is reported. Consequently, the NICE appraisal committee recommended O-Benda+O in the population covered by the marketing authorisation within the Cancer Drug Fund until NICE is able to review the guidance following publication of the final analysis of GADOLIN

Ponatinib for Treating Acute Lymphoblastic Leukaemia: An Evidence Review Group Perspective of a NICE Single Technology Appraisal

As part of its single technology appraisal (STA) process, the UK National Institute for Health and Care Excellence (NICE) invited the manufacturer (Incyte Corporation) of ponatinib (Inclusig®) to submit evidence of its clinical and cost effectiveness for previously treated Philadelphia-chromosome-positive acute lymphoblastic leukaemia (Ph+ ALL) and chronic myeloid leukaemia. This paper focusses on Ph+ ALL. The School of Health and Related Research Technology Appraisal Group at the University of Sheffield was commissioned to act as the independent evidence review group (ERG). This article presents the critical review of the company's submission by the ERG and the outcome of the NICE guidance. The clinical-effectiveness evidence in the company's submission was derived from a phase II, single-arm, open-label, non-comparative study. Given the lack of comparative evidence, a naïve indirect comparison was performed against re-induction chemotherapy comparing major cytogenetic response and complete remission. Best supportive care (BSC) was assumed to produce no disease response. Despite the limited evidence and potential for biases, this study demonstrated that ponatinib was likely to be an effective treatment for patients with Ph+ ALL. The company submitted a state transition model that analysed the incremental cost effectiveness of ponatinib versus re-induction therapy and BSC for the treatment of Ph+ ALL in patients whose disease is resistant to dasatinib, who are intolerant to dasatinib and for whom subsequent treatment with imatinib is not clinically appropriate or who have the threonine-315-isoleucine mutation. This population was further subdivided into those who were suitable for allogeneic stem cell transplant (allo-SCT) and those who were not. The company's revised economic evaluation, following the clarification process, estimated incremental cost-effectiveness ratios (ICERs) in those suitable for allo-SCT of £31,123 per quality-adjusted life-year (QALY) gained for ponatinib compared with re-induction chemotherapy and £26,624 per QALY gained compared with BSC. For those for whom allo-SCT was unsuitable, the company-estimated ICER compared with BSC was £33,954 per QALY gained. Following a critique of the model, the ERG undertook exploratory analyses that, when combined, produced a range in ICERs (due to uncertainty of the most appropriate overall survival function) of dominant (being less expensive and providing more QALYs) to £11,727 per QALY gained compared with re-induction chemotherapy and between £7892 and £31,696 per QALY gained compared with BSC for those in whom allo-SCT was suitable. For those in whom allo-SCT was not suitable, the ERG estimated that ponatinib was dominant. During the consultation period, the company agreed a revised patient access scheme (PAS) that reduced the ICER ranges to £7156 to £29,995 per QALY gained versus BSC and to less than £5000 per QALY gained versus re-induction chemotherapy. In people for whom allo-SCT was unsuitable, ponatinib dominated BSC. The NICE appraisal committee concluded that ponatinib is a cost-effective use of UK NHS resources in the considered population, subject to the company providing the agreed discount in the PAS

Ponatinib for Treating Chronic Myeloid Leukaemia: An Evidence Review Group Perspective of a NICE Single Technology Appraisal

As part of its single technology appraisal process, the National Institute for Health and Care Excellence (NICE) invited the company that manufactures ponatinib (Inclusig®; Incyte Corporation) to submit evidence for the clinical and cost effectiveness for previously treated chronic myeloid leukaemia (CML) and Philadelphia-chromosome-positive acute lymphoblastic leukaemia (Ph+ ALL). This paper focusses on the three phases of CML: the chronic phase (CP), the accelerated phase (AP) and the blast crisis phase (BP). The School of Health and Related Research Technology Appraisal Group at the University of Sheffield was commissioned to act as the independent Evidence Review Group (ERG). This article presents the critical review of the company's submission by the ERG and the outcome of the NICE guidance. Clinical evidence for ponatinib was derived from a phase II, industry-sponsored, single-arm, open-label, multicentre, non-comparative study. Despite the limited evidence and potential for biases, this study demonstrated that ponatinib was likely to be an effective treatment (in terms of major cytogenetic response and major haematological response) with an acceptable safety profile for patients with CML. Given the absence of any head-to-head studies comparing ponatinib with other relevant comparators, the company undertook a matching-adjusted indirect comparison (MAIC) of ponatinib with bosutinib. The approach was only used for patients with CP-CML because comprehensive data were not available for the AP- or BP-CML groups to allow the matching technique to be used. Despite the uncertainty about the MAIC approach, ponatinib was considered likely to offer advantages over bosutinib in the third-line setting, particularly for complete cytogenetic response. The company developed two health economic models to assess the cost effectiveness of ponatinib for the treatment of patients in CP-CML or in advanced CML (AP- or BP-CML, which were modelled separately). The company did not adequately explore the uncertainty in the survivor functions. As a result, the ERG believed the uncertainty in the decision problem was underestimated. Exploratory analyses undertaken by the ERG produced the following results for ponatinib. In CP-CML, from £18,246 to £27,667 per quality-adjusted life-year (QALY) gained compared with best supportive care (BSC), from £19,680 to £37,381 per QALY gained compared with bosutinib and from £18,279 per QALY gained to dominated compared with allogeneic stem cell transplant (allo-SCT). In AP-CML, the cost per QALY gained for ponatinib ranged from £7123 to £17,625 compared with BSC, and from dominating to £61,896 per QALY gained compared with allo-SCT. In BP-CML, the cost effectiveness of ponatinib ranged from £5033 per QALY gained to dominated compared with allo-SCT, although it was likely to be at the more favourable end of this range, and dominant in all scenarios compared with BSC. The NICE appraisal committee concluded that ponatinib is a cost-effective use of NHS resources in the considered population, subject to the company providing the agreed discount in the Patient Access Scheme

The Sheffield Type 1 Diabetes Policy Model

The Sheffield Type 1 Diabetes Policy Model is a patient-level simulation model of type 1 diabetes and its associated complications, which was developed as part of the National Institute for Health Research Dose Adjustment for Normal Eating (DAFNE) research programme. The aim of this paper is to describe the conceptual modelling, model implementation, and model validation phases of the Sheffield Type 1 Diabetes Model development process. The model is highly flexible and has broad potential application to evaluate DAFNE, other diabetes structured education programmes, and other interventions for type 1 diabetes