Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews

BACKGROUND: Meta-research studies investigating methods, systems, and processes designed to improve the efficiency of systematic review workflows can contribute to building an evidence base that can help to increase value and reduce waste in research. This study demonstrates the use of an economic evaluation framework to compare the costs and effects of four variant approaches to identifying eligible studies for consideration in systematic reviews. METHODS: A cost-effectiveness analysis was conducted using a basic decision-analytic model, to compare the relative efficiency of 'safety first', 'double screening', 'single screening' and 'single screening with text mining' approaches in the title-abstract screening stage of a 'case study' systematic review about undergraduate medical education in UK general practice settings. Incremental cost-effectiveness ratios (ICERs) were calculated as the 'incremental cost per citation 'saved' from inappropriate exclusion' from the review. Resource use and effect parameters were estimated based on retrospective analysis of 'review process' meta-data curated alongside the 'case study' review, in conjunction with retrospective simulation studies to model the integrated use of text mining. Unit cost parameters were estimated based on the 'case study' review's project budget. A base case analysis was conducted, with deterministic sensitivity analyses to investigate the impact of variations in values of key parameters. RESULTS: Use of 'single screening with text mining' would have resulted in title-abstract screening workload reductions (base case analysis) of >60 % compared with other approaches. Across modelled scenarios, the 'safety first' approach was, consistently, equally effective and less costly than conventional 'double screening'. Compared with 'single screening with text mining', estimated ICERs for the two non-dominated approaches (base case analyses) ranged from £1975 ('single screening' without a 'provisionally included' code) to £4427 ('safety first' with a 'provisionally included' code) per citation 'saved'. Patterns of results were consistent between base case and sensitivity analyses. CONCLUSIONS: Alternatives to the conventional 'double screening' approach, integrating text mining, warrant further consideration as potentially more efficient approaches to identifying eligible studies for systematic reviews. Comparable economic evaluations conducted using other systematic review datasets are needed to determine the generalisability of these findings and to build an evidence base to inform guidance for review authors

What are children's trusts? Early findings from a national survey

<i>Background:</i> The Children Act 2004 and National Service Framework for Children, Young People and Maternity Services require fuller integration of health, education and social services for children and young people in England and Wales. The UK government supported the establishment of 35 experimental children's trust pathfinders (henceforth called children's trusts) in England. <i>Methods:</i> A questionnaire was completed by managers in all 35 children's trusts a year after their start. Children's trust documents were examined. Census and performance indicators were compared between children's trust areas and the rest of England. <i>Results</i> Children's trust areas had demographic and social characteristics typical of England. All children's trusts aimed to improve health, education and social services by greater managerial and service integration. All had boards representing the three sectors; other agencies’ representation varied. Two-thirds of children's trusts had moved towards pooling budgets in at least some service areas. At this stage in their development, some had prioritized joint procurement or provision of services, with formal managerial structures, while others favoured an informal strategic planning, co-ordination and information sharing approach. The commonest priorities for services development were for disabled children (16 children's trusts), followed by early intervention (11) and mental health services (8). <i>Conclusions:</i> The diverse strategies adopted by these 35 children's trusts during their first year is due to their own characteristics and to the way government strategy developed during this period. Whilst some prioritized organizational development, joint financing and commissioning, and information sharing, others laid more emphasis on mechanisms for bringing front-line professionals closer together. Their experiences are of value to others deciding how best to integrate children's services

Integrating the framing of clinical questions via PICO into the retrieval of medical literature for systematic reviews

The PICO process is a technique used in evidence based practice to frame and answer clinical questions. It involves structuring the question around four types of clinical information: Population, Intervention, Control or comparison and Outcome. The PICO framework is used extensively in the compilation of systematic reviews as the means of framing research questions. However, when a search strategy (comprising of a large Boolean query) is formulated to retrieve studies for inclusion in the review, PICO is offen ignored. This paper evaluates how PICO annotations can be applied and integrated into retrieval to improve the screening of studies for inclusion in systematic reviews. The task is to increase precision while maintaining the high level of recall essential to ensure systematic reviews are representative and unbiased. Our results show that restricting the search strategies to match studies using PICO annotations improves precision, however recall is slightly reduced, when compared to the non-PICO baseline. This can lead to both time and cost savings when compiling systematic reviews

What do we know about the effects of exposure to ‘Low alcohol’ and equivalent product labelling on the amounts of alcohol, food and tobacco people select and consume? A systematic review

Background Explicit labelling of lower strength alcohol products could reduce alcohol consumption by attracting more people to buy and drink such products instead of higher strength ones. Alternatively, it may lead to more consumption due to a ‘self-licensing’ mechanism. Equivalent labelling of food or tobacco (for example “Low fat” or “Low tar”) could influence consumption of those products by similar mechanisms. This systematic review examined the effects of ‘Low alcohol’ and equivalent labelling of alcohol, food and tobacco products on selection, consumption, and perceptions of products among adults. Methods A systematic review was conducted based on Cochrane methods. Electronic and snowball searches identified 26 eligible studies. Evidence from 12 randomised controlled trials (all on food) was assessed for risk of bias, synthesised using random effects meta-analysis, and interpreted in conjunction with evidence from 14 non-randomised studies (one on alcohol, seven on food and six on tobacco). Outcomes assessed were: quantities of the product (i) selected or (ii) consumed (primary outcomes - behaviours), (iii) intentions to select or consume the product, (iv) beliefs associated with it consumption, (v) product appeal, and (vi) understanding of the label (secondary outcomes – cognitions). Results Evidence for impacts on the primary outcomes (i.e. amounts selected or consumed) was overall of very low quality, showing mixed effects, likely to vary by specific label descriptors, products and population characteristics. Overall very low quality evidence suggested that exposure to ‘Low alcohol’ and equivalent labelling on alcohol, food and tobacco products can shift consumer perceptions of products, with the potential to ‘self-licence’ excess consumption. Conclusions Considerable uncertainty remains about the effects of labels denoting low alcohol, and equivalent labels, on alcohol, food and tobacco selection and consumption. Independent, high-quality studies are urgently needed to inform policies on labelling regulations.This report was joint-funded by the Department of Health in England Policy Research Programme (Policy Research Unit in Behaviour and Health (PR-UN-0409-10109)) and an NIHR Senior Investigator Award (NF-SI-0513-10101) held by T. M. Marteau (corresponding author)

Machine learning reduced workload with minimal risk of missing studies: development and evaluation of an RCT classifier for Cochrane Reviews

BACKGROUND: To describe the development, calibration and evaluation of a machine learning classifier designed to reduce study identification workload in Cochrane for producing systematic reviews. METHODS: A machine learning classifier for retrieving RCTs was developed (the ‘Cochrane RCT Classifier’), with the algorithm trained using a dataset of title-abstract records from Embase, manually labelled by the Cochrane Crowd. The classifier was then calibrated using a further dataset of similar records manually labelled by the Clinical Hedges team, aiming for 99% recall. Finally, the recall of the calibrated classifier was evaluated using records of RCTs included in Cochrane Reviews that had abstracts of sufficient length to allow machine classification. RESULTS: The Cochrane RCT Classifier was trained using 280,620 records (20,454 of which reported RCTs). A classification threshold was set using 49,025 calibration records (1,587 of which reported RCTs) and our bootstrap validation found the classifier had recall of 0.99 (95% CI 0.98 to 0.99) and precision of 0.08 (95% CI 0.06 to 0.12) in this dataset. The final, calibrated RCT classifier correctly retrieved 43,783 (99.5%) of 44,007 RCTs included in Cochrane Reviews but missed 224 (0.5%). Older records were more likely to be missed than those more recently published. CONCLUSIONS: The Cochrane RCT Classifier can reduce manual study identification workload for Cochrane reviews, with a very low and acceptable risk of missing eligible RCTs. This classifier now forms part of the Evidence Pipeline, an integrated workflow deployed within Cochrane to help improve the efficiency of the study identification processes that support systematic review production

Cost-effectiveness of Microsoft Academic Graph with machine learning for automated study identification in a living map of coronavirus disease 2019 (COVID-19) research

BACKGROUND: Conventionally, searching for eligible articles to include in systematic reviews and maps of research has relied primarily on information specialists conducting Boolean searches of multiple databases and manually processing the results, including deduplication between these multiple sources. Searching one, comprehensive source, rather than multiple databases, could save time and resources. Microsoft Academic Graph (MAG) is potentially such a source, containing a network graph structure which provides metadata that can be exploited in machine learning processes. Research is needed to establish the relative advantage of using MAG as a single source, compared with conventional searches of multiple databases. This study sought to establish whether: (a) MAG is sufficiently comprehensive to maintain our living map of coronavirus disease 2019 (COVID-19) research; and (b) eligible records can be identified with an acceptably high level of specificity. METHODS: We conducted a pragmatic, eight-arm cost-effectiveness analysis (simulation study) to assess the costs, recall and precision of our semi-automated MAG-enabled workflow versus conventional searches of MEDLINE and Embase (with and without machine learning classifiers, active learning and/or fixed screening targets) for maintaining a living map of COVID-19 research. Resource use data (time use) were collected from information specialists and other researchers involved in map production. RESULTS: MAG-enabled workflows dominated MEDLINE-Embase workflows in both the base case and sensitivity analyses. At one month (base case analysis) our MAG-enabled workflow with machine learning, active learning and fixed screening targets identified n=469 more new, eligible articles for inclusion in our living map – and cost £3,179 GBP ($5,691 AUD) less – than conventional MEDLINE-Embase searches without any automation or fixed screening targets. CONCLUSIONS: MAG-enabled continuous surveillance workflows have potential to revolutionise study identification methods for living maps, specialised registers, databases of research studies and/or collections of systematic reviews, by increasing their recall and coverage, whilst reducing production costs

Issues in the incorporation of economic perspectives and evidence into Cochrane reviews

Methods for systematic reviews of the effects of health interventions have focused mainly on addressing the question of 'What works?' or 'Is this intervention effective in achieving one or more specific outcomes?' Addressing the question 'Is it worth it given the resources available?' has received less attention. This latter question can be addressed by applying an economic lens to the systematic review process.This paper reflects on the value and desire for the consideration by end users for coverage of an economic perspective in a Cochrane review and outlines two potential approaches and future directions

Using automation to produce a ‘living map’ of the COVID-19 research literature

The COVID-19 pandemic has disrupted life worldwide and presented unique challenges in the health evidencesynthesis space. The urgent nature of the pandemic required extreme rapidity for keeping track of research, andthis presented a unique opportunity for long-proposed automation systems to be deployed and evaluated. Wecompared the use of novel automation technologies with conventional manual screening; and Microsoft AcademicGraph (MAG) with the MEDLINE and Embase databases locating the emerging research evidence. We foundthat a new workflow involving machine learning to identify relevant research in MAG achieved a much higherrecall with lower manual effort than using conventional approaches

Improving ranking for systematic reviews using query adaptation

Identifying relevant studies for inclusion in systematic reviews requires significant effort from human experts who manually screen large numbers of studies. The problem is made more difficult by the growing volume of medical literature and Information Retrieval techniques have proved to be useful to reduce workload. Reviewers are often interested in particular types of evidence such as Diagnostic Test Accuracy studies. This paper explores the use of query adaption to identify particular types of evidence and thereby reduce the workload placed on reviewers. A simple retrieval system that ranks studies using TF.IDF weighted cosine similarity was implemented. The Log-Likelihood, ChiSquared and Odds-Ratio lexical statistics and relevance feedback were used to generate sets of terms that indicate evidence relevant to Diagnostic Test Accuracy reviews. Experiments using a set of 80 systematic reviews from the CLEF2017 and CLEF2018 eHealth tasks demonstrate that the approach improves retrieval performance

Living systematic reviews:2. Combining human and machine effort

New approaches to evidence synthesis, which utilise human effort and machine automation in mutually reinforcing ways, can enhance the feasibility and sustainability of living systematic reviews. Human effort is a scarce and valuable resource, required when automation is impossible or undesirable, and includes contributions from online communities ('crowds') as well as more conventional contributions from review authors and information specialists. Automation can assist with some systematic review tasks, including searching, eligibility assessment, identification and retrieval of full text reports, extraction of data, and risk of bias assessment. Workflows can be developed in which human effort and machine automation can each enable the other to operate in more effective and efficient ways, offering substantial enhancement to the productivity of systematic reviews. This paper describes and discusses the potential - and limitations - of new ways of undertaking specific tasks in living systematic reviews, identifying areas where these human / machine 'technologies' are already in use, and where further research and development is needed. While the context is living systematic reviews, many of these enabling technologies apply equally to standard approaches to systematic reviewing