Quantifying the impact and relevance of scientific research

Qualitative and quantitative methods are being developed to measure the impacts of research on society, but they suffer from serious drawbacks associated with linking a piece of research to its subsequent impacts. We have developed a method to derive impact scores for individual research publications according to their contribution to answering questions of quantified importance to end users of research. To demonstrate the approach, here we evaluate the impacts of research into means of conserving wild bee populations in the UK. For published papers, there is a weak positive correlation between our impact score and the impact factor of the journal. The process identifies publications that provide high quality evidence relating to issues of strong concern. It can also be used to set future research agendas

A third of systematic reviews changed or did not specify the primary outcome : A PROSPERO register study

OBJECTIVES: To examine outcome reporting bias of systematic reviews registered in PROSPERO. STUDY DESIGN AND SETTING: Retrospective cohort study. The primary outcomes from systematic review publications were compared with those reported in the corresponding PROSPERO records; discrepancies in the primary outcomes were assessed as upgrades, additions, omissions or downgrades. Relative risks (RR) and 95% confidence intervals (CI) were calculated to determine the likelihood of having a change in primary outcome when the meta-analysis result was favourable and statistically significant. RESULTS: 96 systematic reviews were published. A discrepancy in the primary outcome occurred in 32% of the included reviews and 39% of the reviews did not explicitly specify a primary outcome(s); 6% of the primary outcomes were omitted. There was no significant increased risk of adding/upgrading (RR 2.14, 95% CI 0.53 to 8.63) or decreased risk of downgrading (RR 0.76, 0.27-2.17) an outcome when the meta-analysis result was favourable and statistically significant. As well, there was no significant increased risk of adding/upgrading (RR 0.89, 0.31-2.53) or decreased risk of downgrading (RR 0.56, 0.29-1.08) an outcome when the conclusion was positive. CONCLUSIONS: We recommend review authors carefully consider primary outcome selection and journals are encouraged to focus acceptance on registered systematic reviews

Can electronic search engines optimize screening of search results in systematic reviews: an empirical study

BACKGROUND: Most electronic search efforts directed at identifying primary studies for inclusion in systematic reviews rely on the optimal Boolean search features of search interfaces such as DIALOG(® )and Ovid™. Our objective is to test the ability of an Ultraseek(® )search engine to rank MEDLINE(® )records of the included studies of Cochrane reviews within the top half of all the records retrieved by the Boolean MEDLINE search used by the reviewers. METHODS: Collections were created using the MEDLINE bibliographic records of included and excluded studies listed in the review and all records retrieved by the MEDLINE search. Records were converted to individual HTML files. Collections of records were indexed and searched through a statistical search engine, Ultraseek, using review-specific search terms. Our data sources, systematic reviews published in the Cochrane library, were included if they reported using at least one phase of the Cochrane Highly Sensitive Search Strategy (HSSS), provided citations for both included and excluded studies and conducted a meta-analysis using a binary outcome measure. Reviews were selected if they yielded between 1000–6000 records when the MEDLINE search strategy was replicated. RESULTS: Nine Cochrane reviews were included. Included studies within the Cochrane reviews were found within the first 500 retrieved studies more often than would be expected by chance. Across all reviews, recall of included studies into the top 500 was 0.70. There was no statistically significant difference in ranking when comparing included studies with just the subset of excluded studies listed as excluded in the published review. CONCLUSION: The relevance ranking provided by the search engine was better than expected by chance and shows promise for the preliminary evaluation of large results from Boolean searches. A statistical search engine does not appear to be able to make fine discriminations concerning the relevance of bibliographic records that have been pre-screened by systematic reviewers

An alternative to the hand searching gold standard: validating methodological search filters using relative recall

BACKGROUND: Search filters or hedges play an important role in evidence-based medicine but their development depends on the availability of a "gold standard" – a reference standard against which to establish the performance of the filter. We demonstrate the feasibility of using relative recall of included studies from multiple systematic reviews to validate methodological search filters as an alternative to validation against a gold standard formed through hand searching. METHODS: We identified 105 Cochrane reviews that used the Highly Sensitive Search Strategy (HSSS), included randomized or quasi-randomized controlled trials, and reported their included studies. We measured the ability of two published and one novel variant of the HSSS to retrieve the MEDLINE-index studies included in these reviews. RESULTS: The systematic reviews were comprehensive in their searches. 72% of included primary studies were indexed in MEDLINE. Relative recall of the three strategies ranged from .98 to .91 across all reviews and more comprehensive strategies showed higher recall. CONCLUSION: An approach using relative recall instead of a hand searching gold standard proved feasible and produced recall figures that were congruent with previously published figures for the HSSS. This technique would permit validation of a methodological filter using a collection of approximately 100 studies of the chosen design drawn from the included studies of multiple systematic reviews that used comprehensive search strategies

Outcomes in pediatric studies of medium-chain acyl-coA dehydrogenase (MCAD) deficiency and phenylketonuria (PKU): a review.

BACKGROUND: Inherited metabolic diseases (IMDs) are a group of individually rare single-gene diseases. For many IMDs, there is a paucity of high-quality evidence that evaluates the effectiveness of clinical interventions. Clinical effectiveness trials of IMD interventions could be supported through the development of core outcome sets (COSs), a recommended minimum set of standardized, high-quality outcomes and associated outcome measurement instruments to be incorporated by all trials in an area of study. We began the process of establishing pediatric COSs for two IMDs, medium-chain acyl-CoA dehydrogenase (MCAD) deficiency and phenylketonuria (PKU), by reviewing published literature to describe outcomes reported by authors, identify heterogeneity in outcomes across studies, and assemble a candidate list of outcomes. METHODS: We used a comprehensive search strategy to identify primary studies and guidelines relevant to children with MCAD deficiency and PKU, extracting study characteristics and outcome information from eligible studies including outcome measurement instruments for select outcomes. Informed by an established framework and a previously published pediatric COS, outcomes were grouped into five, mutually-exclusive, a priori core areas: growth and development, life impact, pathophysiological manifestations, resource use, and death. RESULTS: For MCAD deficiency, we identified 83 outcomes from 52 articles. The most frequently represented core area was pathophysiological manifestations, with 33 outcomes reported in 29/52 articles (56%). Death was the most frequently reported outcome. One-third of outcomes were reported by a single study. The most diversely measured outcome was cognition and intelligence/IQ for which eight unique measurement instruments were reported among 14 articles. For PKU, we identified 97 outcomes from 343 articles. The most frequently represented core area was pathophysiological manifestations with 31 outcomes reported in 281/343 articles (82%). Phenylalanine concentration was the most frequently reported outcome. Sixteen percent of outcomes were reported by a single study. Similar to MCAD deficiency, the most diversely measured PKU outcome was cognition and intelligence/IQ with 39 different instruments reported among 82 articles. CONCLUSIONS: Heterogeneity of reported outcomes and outcome measurement instruments across published studies for both MCAD deficiency and PKU highlights the need for COSs for these diseases, to promote the use of meaningful outcomes and facilitate comparisons across studies

Funding source, trial outcome and reporting quality: are they related? Results of a pilot study

BACKGROUND: There has been increasing concern regarding the potential effects of the commercialization of research. METHODS: In order to examine the relationships between funding source, trial outcome and reporting quality, recent issues of five peer-reviewed, high impact factor, general medical journals were hand-searched to identify a sample of 100 randomized controlled trials (20 trials/journal). Relevant data, including funding source (industry/not-for-profit/mixed/not reported) and statistical significance of primary outcome (favouring new treatment/favouring conventional treatment/neutral/unclear), were abstracted. Quality scores were assigned using the Jadad scale and the adequacy of allocation concealment. RESULTS: Sixty-six percent of trials received some industry funding. Trial outcome was not associated with funding source (p= .461). There was a preponderance of favourable statistical conclusions among published trials with 67% reporting results that favored a new treatment whereas 6% favoured the conventional treatment. Quality scores were not associated with funding source or trial outcome. CONCLUSIONS: It is not known whether the absence of significant associations between funding source, trial outcome and reporting quality reflects a true absence of an association or is an artefact of inadequate statistical power, reliance on voluntary disclosure of funding information, a focus on trials recently published in the top medical journals, or some combination thereof. Continued and expanded monitoring of potential conflicts is recommended, particularly in light of new guidelines for disclosure that have been endorsed by the ICMJE

PRISMA Extension for Scoping Reviews (PRISMA-ScR) : Checklist and Explanation

Scoping reviews, a type of knowledge synthesis, follow a systematic approach to map evidence on a topic and identify main concepts, theories, sources, and knowledge gaps. Although more scoping reviews are being done, their methodological and reporting quality need improvement. This document presents the PRISMA-ScR (Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews) checklist and explanation. The checklist was developed by a 24-member expert panel and 2 research leads following published guidance from the EQUATOR (Enhancing the QUAlity and Transparency Of health Research) Network. The final checklist contains 20 essential reporting items and 2 optional items. The authors provide a rationale and an example of good reporting for each item. The intent of the PRISMA-ScR is to help readers (including researchers, publishers, commissioners, policymakers, health care providers, guideline developers, and patients or consumers) develop a greater understanding of relevant terminology, core concepts, and key items to report for scoping reviews

PANC Study (Pancreatitis: A National Cohort Study): national cohort study examining the first 30 days from presentation of acute pancreatitis in the UK

Abstract Background Acute pancreatitis is a common, yet complex, emergency surgical presentation. Multiple guidelines exist and management can vary significantly. The aim of this first UK, multicentre, prospective cohort study was to assess the variation in management of acute pancreatitis to guide resource planning and optimize treatment. Methods All patients aged greater than or equal to 18 years presenting with acute pancreatitis, as per the Atlanta criteria, from March to April 2021 were eligible for inclusion and followed up for 30 days. Anonymized data were uploaded to a secure electronic database in line with local governance approvals. Results A total of 113 hospitals contributed data on 2580 patients, with an equal sex distribution and a mean age of 57 years. The aetiology was gallstones in 50.6 per cent, with idiopathic the next most common (22.4 per cent). In addition to the 7.6 per cent with a diagnosis of chronic pancreatitis, 20.1 per cent of patients had a previous episode of acute pancreatitis. One in 20 patients were classed as having severe pancreatitis, as per the Atlanta criteria. The overall mortality rate was 2.3 per cent at 30 days, but rose to one in three in the severe group. Predictors of death included male sex, increased age, and frailty; previous acute pancreatitis and gallstones as aetiologies were protective. Smoking status and body mass index did not affect death. Conclusion Most patients presenting with acute pancreatitis have a mild, self-limiting disease. Rates of patients with idiopathic pancreatitis are high. Recurrent attacks of pancreatitis are common, but are likely to have reduced risk of death on subsequent admissions. </jats:sec