Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers

BACKGROUND: Citation screening is time consuming and inefficient. We sought to evaluate the performance of Abstrackr, a semi-automated online tool for predictive title and abstract screening. METHODS: Four systematic reviews (aHUS, dietary fibre, ECHO, rituximab) were used to evaluate Abstrackr. Citations from electronic searches of biomedical databases were imported into Abstrackr, and titles and abstracts were screened and included or excluded according to the entry criteria. This process was continued until Abstrackr predicted and classified the remaining unscreened citations as relevant or irrelevant. These classification predictions were checked for accuracy against the original review decisions. Sensitivity analyses were performed to assess the effects of including case reports in the aHUS dataset whilst screening and the effects of using larger imbalanced datasets with the ECHO dataset. The performance of Abstrackr was calculated according to the number of relevant studies missed, the workload saving, the false negative rate, and the precision of the algorithm to correctly predict relevant studies for inclusion, i.e. further full text inspection. RESULTS: Of the unscreened citations, Abstrackr’s prediction algorithm correctly identified all relevant citations for the rituximab and dietary fibre reviews. However, one relevant citation in both the aHUS and ECHO reviews was incorrectly predicted as not relevant. The workload saving achieved with Abstrackr varied depending on the complexity and size of the reviews (9 % rituximab, 40 % dietary fibre, 67 % aHUS, and 57 % ECHO). The proportion of citations predicted as relevant, and therefore, warranting further full text inspection (i.e. the precision of the prediction) ranged from 16 % (aHUS) to 45 % (rituximab) and was affected by the complexity of the reviews. The false negative rate ranged from 2.4 to 21.7 %. Sensitivity analysis performed on the aHUS dataset increased the precision from 16 to 25 % and increased the workload saving by 10 % but increased the number of relevant studies missed. Sensitivity analysis performed with the larger ECHO dataset increased the workload saving (80 %) but reduced the precision (6.8 %) and increased the number of missed citations. CONCLUSIONS: Semi-automated title and abstract screening with Abstrackr has the potential to save time and reduce research waste

Better duplicate detection for systematic reviewers: Evaluation of Systematic Review Assistant-Deduplication Module

BACKGROUND: A major problem arising from searching across bibliographic databases is the retrieval of duplicate citations. Removing such duplicates is an essential task to ensure systematic reviewers do not waste time screening the same citation multiple times. Although reference management software use algorithms to remove duplicate records, this is only partially successful and necessitates removing the remaining duplicates manually. This time-consuming task leads to wasted resources. We sought to evaluate the effectiveness of a newly developed deduplication program against EndNote. METHODS: A literature search of 1,988 citations was manually inspected and duplicate citations identified and coded to create a benchmark dataset. The Systematic Review Assistant-Deduplication Module (SRA-DM) was iteratively developed and tested using the benchmark dataset and compared with EndNote’s default one step auto-deduplication process matching on (‘author’, ‘year’, ‘title’). The accuracy of deduplication was reported by calculating the sensitivity and specificity. Further validation tests, with three additional benchmarked literature searches comprising a total of 4,563 citations were performed to determine the reliability of the SRA-DM algorithm. RESULTS: The sensitivity (84%) and specificity (100%) of the SRA-DM was superior to EndNote (sensitivity 51%, specificity 99.83%). Validation testing on three additional biomedical literature searches demonstrated that SRA-DM consistently achieved higher sensitivity than EndNote (90% vs 63%), (84% vs 73%) and (84% vs 64%). Furthermore, the specificity of SRA-DM was 100%, whereas the specificity of EndNote was imperfect (average 99.75%) with some unique records wrongly assigned as duplicates. Overall, there was a 42.86% increase in the number of duplicates records detected with SRA-DM compared with EndNote auto-deduplication. CONCLUSIONS: The Systematic Review Assistant-Deduplication Module offers users a reliable program to remove duplicate records with greater sensitivity and specificity than EndNote. This application will save researchers and information specialists time and avoid research waste. The deduplication program is freely available online

A comparison of the performance of seven key bibliographic databases in identifying all relevant systematic reviews of interventions for hypertension

BACKGROUND: Bibliographic databases are the primary resource for identifying systematic reviews of health care interventions. Reliable retrieval of systematic reviews depends on the scope of indexing used by database providers. Therefore, searching one database may be insufficient, but it is unclear how many need to be searched. We sought to evaluate the performance of seven major bibliographic databases for the identification of systematic reviews for hypertension. METHODS: We searched seven databases (Cochrane library, Database of Abstracts of Reviews of Effects (DARE), Excerpta Medica Database (EMBASE), Epistemonikos, Medical Literature Analysis and Retrieval System Online (MEDLINE), PubMed Health and Turning Research Into Practice (TRIP)) from 2003 to 2015 for systematic reviews of any intervention for hypertension. Citations retrieved were screened for relevance, coded and checked for screening consistency using a fuzzy text matching query. The performance of each database was assessed by calculating its sensitivity, precision, the number of missed reviews and the number of unique records retrieved. RESULTS: Four hundred systematic reviews were identified for inclusion from 11,381 citations retrieved from seven databases. No single database identified all the retrieved systematic reviews for hypertension. EMBASE identified the most reviews (sensitivity 69 %) but also retrieved the most irrelevant citations with 7.2 % precision (Pr). The sensitivity of the Cochrane library was 60 %, DARE 57 %, MEDLINE 57 %, PubMed Health 53 %, Epistemonikos 49 % and TRIP 33 %. EMBASE contained the highest number of unique records (n = 43). The Cochrane library identified seven unique records and had the highest precision (Pr = 30 %), followed by Epistemonikos (n = 2, Pr = 19 %). No unique records were found in PubMed Health (Pr = 24 %) DARE (Pr = 21 %), TRIP (Pr = 10 %) or MEDLINE (Pr = 10 %). Searching EMBASE and the Cochrane library identified 88 % of all systematic reviews in the reference set, and searching the freely available databases (Cochrane, Epistemonikos, MEDLINE) identified 83 % of all the reviews. The databases were re-analysed after systematic reviews of non-conventional interventions (e.g. yoga, acupuncture) were removed. Similarly, no database identified all the retrieved systematic reviews. EMBASE identified the most relevant systematic reviews (sensitivity 73 %) but also retrieved the most irrelevant citations with Pr = 5 %. The sensitivity of the Cochrane database was 62 %, followed by MEDLINE (60 %), DARE (55 %), PubMed Health (54 %), Epistemonikos (50 %) and TRIP (31 %). The precision of the Cochrane library was the highest (20 %), followed by PubMed Health (Pr = 16 %), DARE (Pr = 13 %), Epistemonikos (Pr = 12 %), MEDLINE (Pr = 6 %), TRIP (Pr = 6 %) and EMBASE (Pr = 5 %). EMBASE contained the most unique records (n = 34). The Cochrane library identified seven unique records. The other databases held no unique records. CONCLUSIONS: The coverage of bibliographic databases varies considerably due to differences in their scope and content. Researchers wishing to identify systematic reviews should not rely on one database but search multiple databases

Resistance decay in individuals after antibiotic exposure in primary care: A systematic review and meta-analysis

Abstract Background Antibiotic resistance is an urgent global problem, but reversibility is poorly understood. We examined the development and decay of bacterial resistance in community patients after antibiotic use. Methods This was a systematic review and meta-analysis. PubMed, EMBASE and CENTRAL (from inception to May 2017) were searched, with forward and backward citation searches of the identified studies. We contacted authors whose data were unclear, and of abstract-only reports, for further information. We considered controlled or times-series studies of patients in the community who were given antibiotics and where the subsequent prevalence of resistant bacteria was measured. Two authors extracted risk of bias and data. The meta-analysis used a fixed-effects model. Results Of 24,492 articles screened, five controlled and 20 time-series studies (total 16,353 children and 1461 adults) were eligible. Resistance in Streptococcus pneumoniae initially increased fourfold after penicillin-class antibiotic exposure [odds ratio (OR) 4.2, 95% confidence interval (CI) 3.5–5.4], but this fell after 1 month (OR 1.7, 95% CI 1.3–2.1). After cephalosporin-class antibiotics, resistance increased (OR 2.2, 95%CI 1.7-2.9); and fell to (OR 1.6, 95% CI 1.2-2.3) at 1 month. After macrolide-class antibiotics, resistance increased (OR 3.8, 95% CI 1.9–7.6) and persisted for 1 month (OR 5.2, 95% CI 2.6–10.3) and 3 months (OR 8.1, 95% CI 4.6–14.2, from controlled studies and OR 2.3, 95% CI 0.6–9.4, from time-series studies). Resistance in Haemophilus influenzae after penicillins was not significantly increased (OR 1.3, 95% CI 0.9–1.9) initially but was at 1 month (OR 3.4, 95% CI 1.5–7.6), falling after 3 months (OR 1.0, 95% CI 0.5–2.2). Data were sparse for cephalosporins and macrolides. Resistance in Enterobacter increased post-exposure (OR 3.2, 95% CI 0.9–10.8, from controlled studies and OR 7.1, 95% CI 4.2–12, from time-series studies], but was lower after 1 month (OR 1.8, 95% CI 0.9–3.6). Conclusions Resistance generally increased soon after antibiotic use. For some antibiotic classes and bacteria, it partially diminished after 1 and 3 months, but longer-term data are lacking and urgently needed. Trial registration PROSPERO CRD42015025499

Cannabis and schizophrenia

BACKGROUND Schizophrenia is a mental illness causing disordered beliefs, ideas and sensations. Many people with schizophrenia smoke cannabis, and it is unclear why a large proportion do so and if the effects are harmful or beneficial. It is also unclear what the best method is to allow people with schizophrenia to alter their cannabis intake. OBJECTIVES To assess the effects of specific psychological treatments for cannabis reduction in people with schizophrenia.To assess the effects of antipsychotics for cannabis reduction in people with schizophrenia.To assess the effects of cannabinoids (cannabis related chemical compounds derived from cannabis or manufactured) for symptom reduction in people with schizophrenia. SEARCH METHODS We searched the Cochrane Schizophrenia Group Trials Register, 12 August 2013, which is based on regular searches of BIOSIS, CINAHL, EMBASE, MEDLINE, PUBMED and PsycINFO.We searched all references of articles selected for inclusion for further relevant trials. We contacted the first author of included studies for unpublished trials or data. SELECTION CRITERIA We included all randomised controlled trials involving cannabinoids and schizophrenia/schizophrenia-like illnesses, which assessed:1) treatments to reduce cannabis use in people with schizophrenia;2) the effects of cannabinoids on people with schizophrenia. DATA COLLECTION AND ANALYSIS We independently inspected citations, selected papers and then re-inspected the studies if there were discrepancies, and extracted data. For dichotomous data we calculated risk ratios (RR) and for continuous data, we calculated mean differences (MD), both with 95% confidence intervals (CI) on an intention-to-treat basis, based on a fixed-effect model. We excluded data if loss to follow-up was greater than 50%. We assessed risk of bias for included studies and used GRADE to rate the quality of the evidence. MAIN RESULTS We identified eight randomised trials, involving 530 participants, which met our selection criteria.For the cannabis reduction studies no one treatment showed superiority for reduction in cannabis use. Overall, data were poorly reported for many outcomes of interest. Our main outcomes of interest were medium-term data for cannabis use, global state, mental state, global functioning, adverse events, leaving the study early and satisfaction with treatment. 1. Reduction in cannabis use: adjunct psychological therapies (specifically about cannabis and psychosis) versus treatment as usualResults from one small study showed people receiving adjunct psychological therapies specifically about cannabis and psychosis were no more likely to reduce their intake than those receiving treatment as usual (n = 54, 1 RCT, MD -0.10, 95% CI -2.44 to 2.24, moderate quality evidence). Results for other main outcomes at medium term were also equivocal. No difference in mental state measured on the PANSS positive were observed between groups (n = 62, 1 RCT, MD -0.30 95% CI -2.55 to 1.95, moderate quality evidence). Nor for the outcome of general functioning measured using the World Health Organization Quality of Life BREF (n = 49, 1 RCT, MD 0.90 95% CI -1.15 to 2.95, moderate quality evidence). No data were reported for the other main outcomes of interest 2. Reduction in cannabis use: adjunct psychological therapy (specifically about cannabis and psychosis) versus adjunct non-specific psychoeducation One study compared specific psychological therapy aimed at cannabis reduction with general psychological therapy. At three-month follow-up, the use of cannabis in the previous four weeks was similar between treatment groups (n = 47, 1 RCT, RR 1.04 95% CI 0.62 to 1.74, moderate quality evidence). Again, at a medium-term follow-up, the average mental state scores from the Brief Pscychiatric Rating Scale-Expanded were similar between groups (n = 47, 1 RCT, MD 3.60 95% CI - 5.61 to 12.81, moderate quality evidence). No data were reported for the other main outcomes of interest: global state, general functioning, adverse events, leaving the study early and satisfaction with treatment. 3. Reduction in cannabis use: antipsychotic versus antipsychotic In a small trial comparing effectiveness of olanzapine versus risperidone for cannabis reduction, there was no difference between groups at medium-term follow-up (n = 16, 1 RCT, RR 1.80 95% CI 0.52 to 6.22, moderate quality evidence). The number of participants leaving the study early at medium term was also similar (n = 28, 1 RCT, RR 0.50 95% CI 0.19 to 1.29, moderate quality evidence). Mental state data were reported, however they were reported within the short term and no difference was observed. No data were reported for global state, general functioning, and satisfaction with treatment.With regards to adverse effects data, no study reported medium-term data. Short-term data were presented but overall, no real differences between treatment groups were observed for adverse effects. 4. Cannabinoid as treatment: cannabidiol versus amisulprideAgain, no data were reported for any of the main outcomes of interest at medium term. There were short-term data reported for mental state using the BPRS and PANSS, no overall differences in mental state were observed between treatment groups. AUTHORS' CONCLUSIONS Results are limited and inconclusive due to the small number and size of randomised controlled trials available and quality of data reporting within these trials. More research is needed to a) explore the effects of adjunct psychological therapy that is specifically about cannabis and psychosis as currently there is no evidence for any novel intervention being better than standard treatment,for those that use cannabis and have schizophrenia b) decide the most effective drug treatment in treating those that use cannabis and have schizophrenia, and c) assess the effectiveness of cannabidiol in treating schizophrenia. Currently evidence is insufficient to show cannabidiol has an antipsychotic effect

Creative credits:a randomized controlled industrial policy experiment

This report examines the results of a pilot study, which used a method of evaluation called randomised control trials (RCTs) to see if a popular business support scheme called Creative Credits worked effectively. The pilot study, which began in Manchester in 2009, was structured so that vouchers, or 'Creative Credits', would be randomly allocated to small and medium-sized businesses applying to invest in creative projects such as developing websites, video production and creative marketing campaigns, to see if they had a real effect on innovation. The research found that the firms who were awarded Creative Credits enjoyed a short-term boost in their innovation and sales growth in the six months following completion of their creative projects. However, the positive effects were not sustained, and after 12 months there was no longer a statistically significant difference between the groups that received the credits and those that didn’t. The report argues that these results would have remained hidden using the normal evaluation methods used by government, and calls for RCTs to be used more widely when evaluating policies to support business growth

Assessing an experimental approach to industrial policy evaluation:Applying RCT+ to the case of Creative Credits

Experimental methods of policy evaluation are well-established in social policy and development eco-nomics but are rare in industrial and innovation policy. In this paper, we consider the arguments forapplying experimental methods to industrial policy measures, and propose an experimental policy eval-uation approach (which we call RCT+). This approach combines the randomised assignment of firmsto treatment and control groups with a longitudinal data collection strategy incorporating quantitativeand qualitative data (so-called mixed methods). The RCT+ approach is designed to provide a causativerather than purely summative evaluation, i.e. to assess both ‘whether’ and ‘how’ programme outcomesare achieved. In this paper, we assess the RCT+ approach through an evaluation of Creative Credits – aUK business-to-business innovation voucher initiative intended to promote new innovation partnershipsbetween SMEs and creative service providers. The results suggest the potential value of the RCT+ approachto industrial policy evaluation, and the benefits of mixed methods and longitudinal data collection