Quantifying the impact and relevance of scientific research

Qualitative and quantitative methods are being developed to measure the impacts of research on society, but they suffer from serious drawbacks associated with linking a piece of research to its subsequent impacts. We have developed a method to derive impact scores for individual research publications according to their contribution to answering questions of quantified importance to end users of research. To demonstrate the approach, here we evaluate the impacts of research into means of conserving wild bee populations in the UK. For published papers, there is a weak positive correlation between our impact score and the impact factor of the journal. The process identifies publications that provide high quality evidence relating to issues of strong concern. It can also be used to set future research agendas

The Short-Term Impact of Ontario's Generic Pricing Reforms

Background: Canadians pay amongst the highest generic drug prices in the world. In July 2010, the province of Ontario enacted a policy that halved reimbursement for generic drugs from the public drug plan, and substantially lowered prices for private purchases. We quantified the impact of this policy on overall generic drug expenditures in the province, and projected the impact in other provinces had they mimicked this pricing change. Methods: We used quarterly prescription generic drug dispensing data from the IMS-Brogan CompuScript Audit. We used the price per unit in both the pre- and post-policy period and two economics price indexes to estimate the expenditure reduction in Ontario. Further, we used the post-policy Ontario prices to estimate the potential reduction in other provinces. Results: We estimate that total expenditure on generic drugs in Ontario during the second half of 2010 was between $181 and$194 million below what would be expected if prices had remained at pre-policy level. Over half of the reduction in spending was due to savings on just 10 generic ingredients. If other provinces had matched Ontario’s prices, their expenditures over during the latter half of 2010 would have been $445 million lower. Discussion: We found that if Ontario’s pricing scheme were adopted nationally, overall spending on generic drugs in Canada would drop at least$1.28 billion annually—a 5 % decrease in total prescription drug expenditure. Other provinces should seriously consider both changes to their generic drug prices and the use of more competitive bulk purchasing policies

A third of systematic reviews changed or did not specify the primary outcome : A PROSPERO register study

OBJECTIVES: To examine outcome reporting bias of systematic reviews registered in PROSPERO. STUDY DESIGN AND SETTING: Retrospective cohort study. The primary outcomes from systematic review publications were compared with those reported in the corresponding PROSPERO records; discrepancies in the primary outcomes were assessed as upgrades, additions, omissions or downgrades. Relative risks (RR) and 95% confidence intervals (CI) were calculated to determine the likelihood of having a change in primary outcome when the meta-analysis result was favourable and statistically significant. RESULTS: 96 systematic reviews were published. A discrepancy in the primary outcome occurred in 32% of the included reviews and 39% of the reviews did not explicitly specify a primary outcome(s); 6% of the primary outcomes were omitted. There was no significant increased risk of adding/upgrading (RR 2.14, 95% CI 0.53 to 8.63) or decreased risk of downgrading (RR 0.76, 0.27-2.17) an outcome when the meta-analysis result was favourable and statistically significant. As well, there was no significant increased risk of adding/upgrading (RR 0.89, 0.31-2.53) or decreased risk of downgrading (RR 0.56, 0.29-1.08) an outcome when the conclusion was positive. CONCLUSIONS: We recommend review authors carefully consider primary outcome selection and journals are encouraged to focus acceptance on registered systematic reviews

Does 3-Day Course of Oral Amoxycillin Benefit Children of Non-Severe Pneumonia with Wheeze: A Multicentric Randomised Controlled Trial

WHO-defined pneumonias, treated with antibiotics, are responsible for a significant proportion of childhood morbidity and mortality in the developing countries. Since substantial proportion pneumonias have a viral etiology, where children are more likely to present with wheeze, there is a concern that currently antibiotics are being over-prescribed for it. Hence the current trial was conducted with the objective to show the therapeutic equivalence of two treatments (placebo and amoxycillin) for children presenting with non-severe pneumonia with wheeze, who have persistent fast breathing after nebulisation with salbutamol, and have normal chest radiograph.This multi-centric, randomised placebo controlled double blind clinical trial intended to investigate equivalent efficacy of placebo and amoxicillin and was conducted in ambulatory care settings in eight government hospitals in India. Participants were children aged 2-59 months of age, who received either oral amoxycillin (31-54 mg/Kg/day, in three divided doses for three days) or placebo, and standard bronchodilator therapy. Primary outcome was clinical failure on or before day- 4.We randomized 836 cases in placebo and 835 in amoxycillin group. Clinical failures occurred in 201 (24.0%) on placebo and 166 (19.9%) on amoxycillin (risk difference 4.2% in favour of antibiotic, 95% CI: 0.2 to 8.1). Adherence for both placebo and amoxycillin was >96% and 98.9% subjects were followed up on day- 4. Clinical failure was associated with (i) placebo treatment (adjusted OR = 1.28, 95% CI: 1.01 to1.62), (ii) excess respiratory rate of >10 breaths per minute (adjusted OR = 1.51, 95% CI: 1.19, 1.92), (iii) vomiting at enrolment (adjusted OR = 1.49, 95% CI: 1.13, 1.96), (iv) history of use of broncho-dilators (adjusted OR = 1.71, 95% CI: 1.30, 2.24) and (v) non-adherence (adjusted OR = 8.06, 95% CI: 4.36, 14.92).Treating children with non-severe pneumonia and wheeze with a placebo is not equivalent to treatment with oral amoxycillin.ClinicalTrials.gov NCT00407394

Randomised, Controlled, Assessor Blind Trial Comparing 4% Dimeticone Lotion with 0.5% Malathion Liquid for Head Louse Infestation

BACKGROUND:Malathion 0.5% has been the most prescribed pediculicide in the United Kingdom for around 10 years, and is widely used in Europe and North America. Anecdotal reports suggest malathion treatments are less effective than formerly, but this has not been confirmed clinically. This study was designed to determine whether malathion is still effective and if 4% dimeticone lotion is a more effective treatment for head louse infestation. METHODOLOGY/PRINCIPAL FINDINGS:We designed this study as an assessor blinded, randomised, controlled, parallel group trial involving 58 children and 15 adults with active head louse infestation. Each participant received two applications 7 days apart of either 4% dimeticone lotion, applied for 8 hours or overnight, or 0.5% malathion liquid applied for 12 hours or overnight. All treatment and check-up visits were conducted in participants' homes. Cure of infestation was defined as no evidence of head lice after the second treatment. Some people were found free from lice but later reinfested. Worst case, intention to treat, analysis found dimeticone was significantly more effective than malathion, with 30/43 (69.8%) participants cured using dimeticone compared with 10/30 (33.3%) using malathion (p<0.01, difference 36.4%, 95% confidence interval 14.7% to 58.2%). Per protocol analysis showed cure rates of 30/39 (76.9%) and 10/29 (34.5%) respectively. Irritant reactions were observed in only two participants, both treated with malathion. CONCLUSIONS/SIGNIFICANCE:We concluded that, although malathion liquid is still effective for some people, dimeticone lotion offers a significantly more effective alternative treatment for most people. TRIAL REGISTRATION:Controlled-Trials.com ISRCTN47755726

An alternative to the hand searching gold standard: validating methodological search filters using relative recall

BACKGROUND: Search filters or hedges play an important role in evidence-based medicine but their development depends on the availability of a "gold standard" – a reference standard against which to establish the performance of the filter. We demonstrate the feasibility of using relative recall of included studies from multiple systematic reviews to validate methodological search filters as an alternative to validation against a gold standard formed through hand searching. METHODS: We identified 105 Cochrane reviews that used the Highly Sensitive Search Strategy (HSSS), included randomized or quasi-randomized controlled trials, and reported their included studies. We measured the ability of two published and one novel variant of the HSSS to retrieve the MEDLINE-index studies included in these reviews. RESULTS: The systematic reviews were comprehensive in their searches. 72% of included primary studies were indexed in MEDLINE. Relative recall of the three strategies ranged from .98 to .91 across all reviews and more comprehensive strategies showed higher recall. CONCLUSION: An approach using relative recall instead of a hand searching gold standard proved feasible and produced recall figures that were congruent with previously published figures for the HSSS. This technique would permit validation of a methodological filter using a collection of approximately 100 studies of the chosen design drawn from the included studies of multiple systematic reviews that used comprehensive search strategies

Can electronic search engines optimize screening of search results in systematic reviews: an empirical study

BACKGROUND: Most electronic search efforts directed at identifying primary studies for inclusion in systematic reviews rely on the optimal Boolean search features of search interfaces such as DIALOG(® )and Ovid™. Our objective is to test the ability of an Ultraseek(® )search engine to rank MEDLINE(® )records of the included studies of Cochrane reviews within the top half of all the records retrieved by the Boolean MEDLINE search used by the reviewers. METHODS: Collections were created using the MEDLINE bibliographic records of included and excluded studies listed in the review and all records retrieved by the MEDLINE search. Records were converted to individual HTML files. Collections of records were indexed and searched through a statistical search engine, Ultraseek, using review-specific search terms. Our data sources, systematic reviews published in the Cochrane library, were included if they reported using at least one phase of the Cochrane Highly Sensitive Search Strategy (HSSS), provided citations for both included and excluded studies and conducted a meta-analysis using a binary outcome measure. Reviews were selected if they yielded between 1000–6000 records when the MEDLINE search strategy was replicated. RESULTS: Nine Cochrane reviews were included. Included studies within the Cochrane reviews were found within the first 500 retrieved studies more often than would be expected by chance. Across all reviews, recall of included studies into the top 500 was 0.70. There was no statistically significant difference in ranking when comparing included studies with just the subset of excluded studies listed as excluded in the published review. CONCLUSION: The relevance ranking provided by the search engine was better than expected by chance and shows promise for the preliminary evaluation of large results from Boolean searches. A statistical search engine does not appear to be able to make fine discriminations concerning the relevance of bibliographic records that have been pre-screened by systematic reviewers

Funding source, trial outcome and reporting quality: are they related? Results of a pilot study

BACKGROUND: There has been increasing concern regarding the potential effects of the commercialization of research. METHODS: In order to examine the relationships between funding source, trial outcome and reporting quality, recent issues of five peer-reviewed, high impact factor, general medical journals were hand-searched to identify a sample of 100 randomized controlled trials (20 trials/journal). Relevant data, including funding source (industry/not-for-profit/mixed/not reported) and statistical significance of primary outcome (favouring new treatment/favouring conventional treatment/neutral/unclear), were abstracted. Quality scores were assigned using the Jadad scale and the adequacy of allocation concealment. RESULTS: Sixty-six percent of trials received some industry funding. Trial outcome was not associated with funding source (p= .461). There was a preponderance of favourable statistical conclusions among published trials with 67% reporting results that favored a new treatment whereas 6% favoured the conventional treatment. Quality scores were not associated with funding source or trial outcome. CONCLUSIONS: It is not known whether the absence of significant associations between funding source, trial outcome and reporting quality reflects a true absence of an association or is an artefact of inadequate statistical power, reliance on voluntary disclosure of funding information, a focus on trials recently published in the top medical journals, or some combination thereof. Continued and expanded monitoring of potential conflicts is recommended, particularly in light of new guidelines for disclosure that have been endorsed by the ICMJE

Outcomes in pediatric studies of medium-chain acyl-coA dehydrogenase (MCAD) deficiency and phenylketonuria (PKU): a review.

BACKGROUND: Inherited metabolic diseases (IMDs) are a group of individually rare single-gene diseases. For many IMDs, there is a paucity of high-quality evidence that evaluates the effectiveness of clinical interventions. Clinical effectiveness trials of IMD interventions could be supported through the development of core outcome sets (COSs), a recommended minimum set of standardized, high-quality outcomes and associated outcome measurement instruments to be incorporated by all trials in an area of study. We began the process of establishing pediatric COSs for two IMDs, medium-chain acyl-CoA dehydrogenase (MCAD) deficiency and phenylketonuria (PKU), by reviewing published literature to describe outcomes reported by authors, identify heterogeneity in outcomes across studies, and assemble a candidate list of outcomes. METHODS: We used a comprehensive search strategy to identify primary studies and guidelines relevant to children with MCAD deficiency and PKU, extracting study characteristics and outcome information from eligible studies including outcome measurement instruments for select outcomes. Informed by an established framework and a previously published pediatric COS, outcomes were grouped into five, mutually-exclusive, a priori core areas: growth and development, life impact, pathophysiological manifestations, resource use, and death. RESULTS: For MCAD deficiency, we identified 83 outcomes from 52 articles. The most frequently represented core area was pathophysiological manifestations, with 33 outcomes reported in 29/52 articles (56%). Death was the most frequently reported outcome. One-third of outcomes were reported by a single study. The most diversely measured outcome was cognition and intelligence/IQ for which eight unique measurement instruments were reported among 14 articles. For PKU, we identified 97 outcomes from 343 articles. The most frequently represented core area was pathophysiological manifestations with 31 outcomes reported in 281/343 articles (82%). Phenylalanine concentration was the most frequently reported outcome. Sixteen percent of outcomes were reported by a single study. Similar to MCAD deficiency, the most diversely measured PKU outcome was cognition and intelligence/IQ with 39 different instruments reported among 82 articles. CONCLUSIONS: Heterogeneity of reported outcomes and outcome measurement instruments across published studies for both MCAD deficiency and PKU highlights the need for COSs for these diseases, to promote the use of meaningful outcomes and facilitate comparisons across studies

The Chemokine Receptor CXCR4 Strongly Promotes Neuroblastoma Primary Tumour and Metastatic Growth, but not Invasion

Neuroblastoma (NB) is a heterogeneous, and particularly malignant childhood neoplasm in its higher stages, with a propensity to form metastasis in selected organs, in particular liver and bone marrow, and for which there is still no efficient treatment available beyond surgery. Recent evidence indicates that the CXCR4/CXCL12 chemokine/receptor axis may be involved in promoting NB invasion and metastasis. In this study, we explored the potential role of CXCR4 in the malignant behaviour of NB, using a combination of in vitro functional analyses and in vivo growth and metastasis assessment in an orthotopic NB mouse model. We show here that CXCR4 overexpression in non-metastatic CXCR4-negative NB cells IGR-NB8 and in moderately metastatic, CXCR4 expressing NB cells IGR-N91, strongly increased tumour growth of primary tumours and liver metastases, without altering the frequency or the pattern of metastasis. Moreover shRNA-mediated knock-down experiments confirmed our observations by showing that silencing CXCR4 in NB cells impairs in vitro and almost abrogates in vivo growth. High levels of CXCL12 were detected in the mouse adrenal gland (the primary tumour site), and in the liver suggesting a paracrine effect of host-derived CXCL12 on NB growth. In conclusion, this study reveals a yet unreported NB-specific predominant growth and survival-promoting role of CXCR4, which warrants a critical reconsideration of the role of CXCR4 in the malignant behaviour of NB and other cancers