Meta-analysis: Neither quick nor easy

BACKGROUND: Meta-analysis is often considered to be a simple way to summarize the existing literature. In this paper we describe how a meta-analysis resembles a conventional study, requiring a written protocol with design elements that parallel those of a record review. METHODS: The paper provides a structure for creating a meta-analysis protocol. Some guidelines for measurement of the quality of papers are given. A brief overview of statistical considerations is included. Four papers are reviewed as examples. The examples generally followed the guidelines we specify in reporting the studies and results, but in some of the papers there was insufficient information on the meta-analysis process. CONCLUSIONS: Meta-analysis can be a very useful method to summarize data across many studies, but it requires careful thought, planning and implementation

Big data and data repurposing – using existing data to answer new questions in vascular dementia research

Introduction: Traditional approaches to clinical research have, as yet, failed to provide effective treatments for vascular dementia (VaD). Novel approaches to collation and synthesis of data may allow for time and cost efficient hypothesis generating and testing. These approaches may have particular utility in helping us understand and treat a complex condition such as VaD. Methods: We present an overview of new uses for existing data to progress VaD research. The overview is the result of consultation with various stakeholders, focused literature review and learning from the group’s experience of successful approaches to data repurposing. In particular, we benefitted from the expert discussion and input of delegates at the 9th International Congress on Vascular Dementia (Ljubljana, 16-18th October 2015). Results: We agreed on key areas that could be of relevance to VaD research: systematic review of existing studies; individual patient level analyses of existing trials and cohorts and linking electronic health record data to other datasets. We illustrated each theme with a case-study of an existing project that has utilised this approach. Conclusions: There are many opportunities for the VaD research community to make better use of existing data. The volume of potentially available data is increasing and the opportunities for using these resources to progress the VaD research agenda are exciting. Of course, these approaches come with inherent limitations and biases, as bigger datasets are not necessarily better datasets and maintaining rigour and critical analysis will be key to optimising data use

An assessment of the quality of randomised controlled trials conducted in China

Background: Despite the rapid increase in research in China, little is known about the quality of clinical trials conducted there.Methods: A systematic review and critical appraisal of randomised controlled trials (RCTs) conducted in China and published in 2004 was undertaken to describe their characteristics, assess the quality of their reporting, and where possible, the quality of their conduct. Randomised controlled trials in all disease areas and types of interventions, which took place in China and included Chinese citizens were identified using PubMed and hand searching the Journal Series of the Chinese Medical Association. Quality was assessed against a subset of criteria adapted from the CONSORT statement.Results: Three hundred and seven RCTs were included. One hundred and ninety-nine (64.8%) failed to report methods of randomization and 254 (82.4%) did not mention blinding of either participants or investigators. Reporting of baseline characteristics, primary outcome and length of follow-up was inadequate in a substantial proportion of studies. Fewer than 11% of RCTs mentioned ethical approval and only 18.0% adequately discussed informed consent. However, dropout rates were very favourable with nearly 44% of trials reporting a zero dropout rate.Conclusion: Reporting of RCTs in China requires substantial improvement to meet the targets of the CONSORT statement. The conduct of Chinese RCTs cannot be directly inferred from the standard of reporting; however without good reporting the methods of the trials cannot be clearly ascertained

Effects of the search technique on the measurement of the change in quality of randomized controlled trials over time in the field of brain injury

BACKGROUND: To determine if the search technique that is used to sample randomized controlled trial (RCT) manuscripts from a field of medical science can influence the measurement of the change in quality over time in that field. METHODS: RCT manuscripts in the field of brain injury were identified using two readily-available search techniques: (1) a PubMed MEDLINE search, and (2) the Cochrane Injuries Group (CIG) trials registry. Seven criteria of quality were assessed in each manuscript and related to the year-of-publication of the RCT manuscripts by regression analysis. RESULTS: No change in the frequency of reporting of any individual quality criterion was found in the sample of RCT manuscripts identified by the PubMed MEDLINE search. In the RCT manuscripts of the CIG trials registry, three of the seven criteria showed significant or near-significant increases over time. CONCLUSIONS: We demonstrated that measuring the change in quality over time of a sample of RCT manuscripts from the field of brain injury can be greatly affected by the search technique. This poorly recognized factor may make measurements of the change in RCT quality over time within a given field of medical science unreliable

Additive scales in degenerative disease - calculation of effect sizes and clinical judgment

<p>Abstract</p> <p>Background</p> <p>The therapeutic efficacy of an intervention is often assessed in clinical trials by scales measuring multiple diverse activities that are added to produce a cumulative global score. Medical communities and health care systems subsequently use these data to calculate pooled effect sizes to compare treatments. This is done because major doubt has been cast over the clinical relevance of statistically significant findings relying on <it>p </it>values with the potential to report chance findings. Hence in an aim to overcome this pooling the results of clinical studies into a meta-analyses with a statistical calculus has been assumed to be a more definitive way of deciding of efficacy.</p> <p>Methods</p> <p>We simulate the therapeutic effects as measured with additive scales in patient cohorts with different disease severity and assess the limitations of an effect size calculation of additive scales which are proven mathematically.</p> <p>Results</p> <p>We demonstrate that the major problem, which cannot be overcome by current numerical methods, is the complex nature and neurobiological foundation of clinical psychiatric endpoints in particular and additive scales in general. This is particularly relevant for endpoints used in dementia research. 'Cognition' is composed of functions such as memory, attention, orientation and many more. These individual functions decline in varied and non-linear ways. Here we demonstrate that with progressive diseases cumulative values from multidimensional scales are subject to distortion by the limitations of the additive scale. The non-linearity of the decline of function impedes the calculation of effect sizes based on cumulative values from these multidimensional scales.</p> <p>Conclusions</p> <p>Statistical analysis needs to be guided by boundaries of the biological condition. Alternatively, we suggest a different approach avoiding the error imposed by over-analysis of cumulative global scores from additive scales.</p

The challenges faced in the design, conduct and analysis of surgical randomised controlled trials

Randomised evaluations of surgical interventions are rare; some interventions have been widely adopted without rigorous evaluation. Unlike other medical areas, the randomised controlled trial (RCT) design has not become the default study design for the evaluation of surgical interventions. Surgical trials are difficult to successfully undertake and pose particular practical and methodological challenges. However, RCTs have played a role in the assessment of surgical innovations and there is scope and need for greater use. This article will consider the design, conduct and analysis of an RCT of a surgical intervention. The issues will be reviewed under three headings: the timing of the evaluation, defining the research question and trial design issues. Recommendations on the conduct of future surgical RCTs are made. Collaboration between research and surgical communities is needed to address the distinct issues raised by the assessmentof surgical interventions and enable the conduct of appropriate and well-designed trials.The Health Services Research Unit is funded by the Scottish Government Health DirectoratesPeer reviewedPublisher PD

Obstetric anal sphincter injury: a systematic review of information available on the internet.

OBJECTIVE: There is no systematic evaluation of online health information pertaining to obstetric anal sphincter injury. Therefore, we evaluated the accuracy, credibility, reliability, and readability of online information concerning obstetric anal sphincter injury. MATERIALS AND METHODS: Multiple search engines were searched. The first 30 webpages were identified for each keyword and considered eligible if they provided information regarding obstetric anal sphincter injury. Eligible webpages were assessed by two independent researchers for accuracy (prioritised criteria based upon the RCOG Third and Fourth Degree Tear guideline); credibility; reliability; and readability. RESULTS: Fifty-eight webpages were included. Seventeen webpages (30%) had obtained Health On the Net certification, or Information Standard approval and performed better than those without such approvals (p = 0.039). The best overall performing website was http://www.pat.nhs.uk (score of 146.7). A single webpage (1%) fulfilled the entire criteria for accuracy with a score of 18: www.tamesidehospital.nhs.uk . Twenty-nine webpages (50%) were assessed as credible (scores ≥7). A single webpage achieved a maximum credibility score of 10: www.meht.nhs.uk . Over a third (21 out of 58) were rated as poor or very poor. The highest scoring webpage was http://www.royalsurrey.nhs.uk (score 62). No webpage met the recommended Flesch Reading Ease Score above 70. The intra-class coefficient between researchers was 0.98 (95% CI 0.96-0.99) and 0.94 (95% CI 0.89-0.96) for accuracy and reliability assessments. CONCLUSION: Online information concerning obstetric anal sphincter injury often uses language that is inappropriate for a lay audience and lacks sufficient accuracy, credibility, and reliability

Publication bias in gastroenterological research – a retrospective cohort study based on abstracts submitted to a scientific meeting

BACKGROUND: The aim of this study was to examine the determinants of publication and whether publication bias occurred in gastroenterological research. METHODS: A random sample of abstracts submitted to DDW, the major GI meeting (1992–1995) was evaluated. The publication status was determined by database searches, complemented by a mailed survey to abstract authors. Determinants of publication were examined by Cox proportional hazards model and multiple logistic regression. RESULTS: The sample included abstracts on 326 controlled clinical trials (CCT), 336 other clinical research reports (OCR), and 174 basic science studies (BSS). 392 abstracts (47%) were published as full papers. Acceptance for presentation at the meeting was a strong predictor of subsequent publication for all research types (overall, 54% vs. 34%, OR 2.3, 95% CI 1.7 to 3.1). In the multivariate analysis, multi-center status was found to predict publication (OR 2.8, 95% CI 1.6–4.9). There was no significant association between direction of study results and subsequent publication. Studies were less likely to be published in high impact journals if the results were not statistically significant (OR 0.5, 95 CI 95% 0.3–0.6). The author survey identified lack of time or interest as the main reason for failure to publish. CONCLUSIONS: Abstracts which were selected for presentation at the DDW are more likely to be followed by full publications. The statistical significance of the study results was not found to be a predictor of publication but influences the chances for high impact publication

The interpretation of systematic reviews with meta-analyses: an objective or subjective process?

<p>Abstract</p> <p>Background</p> <p>Discrepancies between the conclusions of different meta-analyses (quantitative syntheses of systematic reviews) are often ascribed to methodological differences. The objective of this study was to determine the discordance in interpretations when meta-analysts are presented with identical data.</p> <p>Methods</p> <p>We searched the literature for all randomized clinical trials (RCT) and review articles on the efficacy of intravenous magnesium in the early post-myocardial infarction period. We organized the articles chronologically and grouped them in packages. The first package included the first RCT, and a summary of the review articles published prior to first RCT. The second package contained the second and third RCT, a meta-analysis based on the data, and a summary of all review articles published prior to the third RCT. Similar packages were created for the 5^thRCT, 10^thRCT, 20^thRCT and 23^rdRCT (all articles). We presented the packages one at a time to eight different reviewers and asked them to answer three clinical questions after each package based solely on the information provided. The clinical questions included whether 1) they believed magnesium is now proven beneficial, 2) they believed magnesium will eventually be proven to be beneficial, and 3) they would recommend its use at this time.</p> <p>Results</p> <p>There was considerable disagreement among the reviewers for each package, and for each question. The discrepancies increased when the heterogeneity of the data increased. In addition, some reviewers became more sceptical of the effectiveness of magnesium over time, and some reviewers became less sceptical.</p> <p>Conclusion</p> <p>The interpretation of the results of systematic reviews with meta-analyses includes a subjective component that can lead to discordant conclusions that are independent of the methodology used to obtain or analyse the data.</p