Does updating improve the methodological and reporting quality of systematic reviews?

BACKGROUND: Systematic reviews (SRs) must be of high quality. The purpose of our research was to compare the methodological and reporting quality of original versus updated Cochrane SRs to determine whether updating had improved these two quality dimensions. METHODS: We identifed updated Cochrane SRs published in issue 4, 2002 of the Cochrane Library. We assessed the updated and original versions of the SRs using two instruments: the 10 item enhanced Overview Quality Assessment Questionnaire (OQAQ), and an 18-item reporting quality checklist and flow chart based upon the Quality of Reporting of Meta-analyses (QUOROM) statement. At least two reviewers extracted data and assessed quality. We calculated the percentage (with a 95% confidence interval) of 'yes' answers to each question. We calculated mean differences in percentage, 95% confidence intervals and p-values for each of the individual items and the overall methodological quality score of the updated and pre-updated versions using OQAQ. RESULTS: We assessed 53 SRs. There was no significant improvement in the global quality score of the OQAQ (mean difference 0.11 (-0.28; 0.70 p = 0.52)). Updated reviews showed a significant improvement of 18.9 (7.2; 30.6 p < .01) on the OQAQ item assessing whether the conclusions drawn by the author(s) were supported by the data and/or analysis presented in the SR. The QUOROM statement showed that the quality of reporting of Cochrane reviews improved in some areas with updating. Improvements were seen on the items relating to data sources reported in the abstract, with a significant difference of 17.0 (9.8; 28.7 p = 0.01), review methods, reported in the abstract 35 (24.1; 49.1 p = 0.00), searching methods 18.9 (9.7; 31.6 p = 0.01), and data abstraction 18.9 (11.7; 30.9 p = 0.00). CONCLUSION: The overall quality of Cochrane SRs is fair-to-good. Although reporting quality improved on certain individual items there was no overall improvement seen with updating and methodological quality remained unchanged. Further improvement of quality of reporting is possible. There is room for improvement of methodological quality as well. Authors updating reviews should address identified methodological or reporting weaknesses. We recommend to give full attention to both quality domains when updating SRs

Reporting of Methodologic Information on Trial Registries for Quality Assessment: A Study of Trial Records Retrieved from the WHO Search Portal

Background: Although randomized clinical trials (RCTs) are considered the gold standard of evidence, their reporting is often suboptimal. Trial registries have the potential to contribute important methodologic information for critical appraisal of study results. Methods and Findings: The objective of the study was to evaluate the reporting of key methodologic study characteristics in trial registries. We identified a random sample (n = 265) of actively recruiting RCTs using the World Health Organization International Clinical Trials Registry Platform (ICTRP) search portal in 2008. We assessed the reporting of relevant domains from the Cochrane Collaboration’s ‘Risk of bias’ tool and other key methodological aspects. Our primary outcomes were the proportion of registry records with adequate reporting of random sequence generation, allocation concealment, blinding, and trial outcomes. Two reviewers independently assessed each record. Weighted overall proportions in the ICTRP search portal for adequate reporting of sequence generation, allocation concealment, blinding (including and excluding open label RCT) and primary outcomes were 5.7% (95% CI 3.0–8.4%), 1.4% (0–2.8%), 41% (35–47%), 8.4% (4.1–13%), and 66% (60–72%), respectively. The proportion of adequately reported RCTs was higher for registries that used specific methodological fields for describing methods of randomization and allocation concealment compared to registries that did not. Concerning other key methodological aspects, weighted overall proportions of RCTs with adequately reported items were as follows: eligibility criteria (81%), secondary outcomes (46%), harm (5%) follow-up duration (62%), description of the interventions (53%) and sample size calculation (1%). Conclusions: Trial registries currently contain limited methodologic information about registered RCTs. In order to permit adequate critical appraisal of trial results reported in journals and registries, trial registries should consider requesting details on key RCT methods to complement journal publications. Full protocols remain the most comprehensive source of methodologic information and should be made publicly available

The Quality of Registration of Clinical Trials

BACKGROUND: Lack of transparency in clinical trial conduct, publication bias and selective reporting bias are still important problems in medical research. Through clinical trials registration, it should be possible to take steps towards resolving some of these problems. However, previous evaluations of registered records of clinical trials have shown that registered information is often incomplete and non-meaningful. If these studies are accurate, this negates the possible benefits of registration of clinical trials. METHODS AND FINDINGS: A 5% sample of records of clinical trials that were registered between 17 June 2008 and 17 June 2009 was taken from the International Clinical Trials Registry Platform (ICTRP) database and assessed for the presence of contact information, the presence of intervention specifics in drug trials and the quality of primary and secondary outcome reporting. 731 records were included. More than half of the records were registered after recruitment of the first participant. The name of a contact person was available in 94.4% of records from non-industry funded trials and 53.7% of records from industry funded trials. Either an email address or a phone number was present in 76.5% of non-industry funded trial records and in 56.5% of industry funded trial records. Although a drug name or company serial number was almost always provided, other drug intervention specifics were often omitted from registration. Of 3643 reported outcomes, 34.9% were specific measures with a meaningful time frame. CONCLUSIONS: Clinical trials registration has the potential to contribute substantially to improving clinical trial transparency and reducing publication bias and selective reporting. These potential benefits are currently undermined by deficiencies in the provision of information in key areas of registered records

Routine Antenatal Anti-D Prophylaxis in Women Who Are Rh(D) Negative: Meta-Analyses Adjusted for Differences in Study Design and Quality

Background: To estimate the effectiveness of routine antenatal anti-D prophylaxis for preventing sensitisation in pregnant Rhesus negative women, and to explore whether this depends on the treatment regimen adopted. Methods: Ten studies identified in a previous systematic literature search were included. Potential sources of bias were systematically identified using bias checklists, and their impact and uncertainty were quantified using expert opinion. Study results were adjusted for biases and combined, first in a random-effects meta-analysis and then in a random-effects metaregression analysis. Results: In a conventional meta-analysis, the pooled odds ratio for sensitisation was estimated as 0.25 (95 % CI 0.18, 0.36), comparing routine antenatal anti-D prophylaxis to control, with some heterogeneity (I 2 = 19%). However, this naïve analysis ignores substantial differences in study quality and design. After adjusting for these, the pooled odds ratio for sensitisation was estimated as 0.31 (95 % CI 0.17, 0.56), with no evidence of heterogeneity (I 2 = 0%). A meta-regression analysis wa

Recommendations by Cochrane Review Groups for assessment of the risk of bias in studies

<p>Abstract</p> <p>Background</p> <p>Assessing the risk of bias in individual studies in a systematic review can be done using individual components or by summarizing the study quality in an overall score.</p> <p>Methods</p> <p>We examined the instructions to authors of the 50 Cochrane Review Groups that focus on clinical interventions for recommendations on methodological quality assessment of studies.</p> <p>Results</p> <p>Forty-one of the review groups (82%) recommended quality assessment using components and nine using a scale. All groups recommending components recommended to assess concealment of allocation, compared to only two of the groups recommending scales (P < 0.0001). Thirty-five groups (70%) recommended assessment of sequence generation and 21 groups (42%) recommended assessment of intention-to-treat analysis. Only 28 groups (56%) had specific recommendations for using the quality assessment of studies analytically in reviews, with sensitivity analysis, quality as an inclusion threshold and subgroup analysis being the most commonly recommended methods. The scales recommended had problems in the individual items and some of the groups recommending components recommended items not related to bias in their quality assessment.</p> <p>Conclusion</p> <p>We found that recommendations by some groups were not based on empirical evidence and many groups had no recommendations on how to use the quality assessment in reviews. We suggest that all Cochrane Review Groups refer to the Cochrane Handbook for Systematic Reviews of Interventions, which is evidence-based, in their instructions to authors and that their own guidelines are kept to a minimum and describe only how methodological topics that are specific to their fields should be handled.</p

Completeness and Changes in Registered Data and Reporting Bias of Randomized Controlled Trials in ICMJE Journals after Trial Registration Policy

We assessed the adequacy of randomized controlled trial (RCT) registration, changes to registration data and reporting completeness for articles in ICMJE journals during 2.5 years after registration requirement policy.For a set of 149 reports of 152 RCTs with ClinicalTrials.gov registration number, published from September 2005 to April 2008, we evaluated the completeness of 9 items from WHO 20-item Minimum Data Set relevant for assessing trial quality. We also assessed changes to the registration elements at the Archive site of ClinicalTrials.gov and compared published and registry data.RCTs were mostly registered before 13 September 2005 deadline (n = 101, 66.4%); 118 (77.6%) started recruitment before and 31 (20.4%) after registration. At the time of registration, 152 RCTs had a total of 224 missing registry fields, most commonly 'Key secondary outcomes' (44.1% RCTs) and 'Primary outcome' (38.8%). More RCTs with post-registration recruitment had missing Minimum Data Set items than RCTs with pre-registration recruitment: 57/118 (48.3%) vs. 24/31 (77.4%) (χ(2) (1) = 7.255, P = 0.007). Major changes in the data entries were found for 31 (25.2%) RCTs. The number of RCTs with differences between registered and published data ranged from 21 (13.8%) for Study type to 118 (77.6%) for Target sample size.ICMJE journals published RCTs with proper registration but the registration data were often not adequate, underwent substantial changes in the registry over time and differed in registered and published data. Editors need to establish quality control procedures in the journals so that they continue to contribute to the increased transparency of clinical trials

Applicability and generalisability of the results of systematic reviews to public health practice and policy: a systematic review

<p>Abstract</p> <p>Background</p> <p>The purpose of the study was to evaluate systematic reviews of research into two public health priorities, tobacco consumption and HIV infection, in terms of the reporting of data related to the applicability of trial results (i.e., whether the results of a trial can be reasonably applied or generalized to a definable group of patients in a particular setting in routine practice, also called external validity or generalisability).</p> <p>Methods</p> <p>All systematic reviews of interventions aimed at reducing or stopping tobacco use and treating or preventing HIV infection published in the Cochrane database of systematic reviews and in journals indexed in MEDLINE between January 1997 and December 2007 were selected. We used a standardized data abstraction form to extract data related to applicability in terms of the context of the trial, (country, centres, settings), participants (recruitment, inclusion and exclusion criteria, baseline characteristics of participants such as age, sex, ethnicity, coexisting diseases or co-morbidities, and socioeconomic status), treatment (duration, intensity/dose of treatment, timing and delivery format), and the outcomes assessment from selected reviews.</p> <p>Results</p> <p>A total of 98 systematic reviews were selected (57 Cochrane reviews and 41 non-Cochrane reviews); 49 evaluated interventions aimed at reducing or stopping tobacco use and 49 treating or preventing HIV infection. The setting of the individual studies was reported in 45 (46%) of the systematic reviews, the number of centres in 21 (21%), and the country where the trial took place in 62 (63%). Inclusion and exclusion criteria of the included studies were reported in 16 (16%) and 13 (13%) of the reviews, respectively. Baseline characteristics of participants in the included studies were described in 59 (60%) of the reviews. These characteristics concerned age in about half of the reviews, sex in 46 (47%), and ethnicity in 9 (9%).</p> <p>Applicability of results was discussed in 13 (13%) of the systematic reviews. The reporting was better in systematic reviews by the Cochrane Collaboration than by non-Cochrane groups.</p> <p>Conclusions</p> <p>Our study highlighted the lack of consideration of applicability of results in systematic reviews of research into 2 public health priorities: tobacco consumption and HIV infection.</p

Disentangling manual muscle testing and Applied Kinesiology: critique and reinterpretation of a literature review

Cuthbert and Goodheart recently published a narrative review on the reliability and validity of manual muscle testing (MMT) in the Journal. The authors should be recognized for their effort to synthesize this vast body of literature. However, the review contains critical errors in the search methods, inclusion criteria, quality assessment, validity definitions, study interpretation, literature synthesis, generalizability of study findings, and conclusion formulation that merit a reconsideration of the authors' findings. Most importantly, a misunderstanding of the review could easily arise because the authors did not distinguish the general use of muscle strength testing from the specific applications that distinguish the Applied Kinesiology (AK) chiropractic technique. The article makes the fundamental error of implying that the reliability and validity of manual muscle testing lends some degree of credibility to the unique diagnostic procedures of AK. The purpose of this commentary is to provide a critical appraisal of the review, suggest conclusions consistent with the literature both reviewed and omitted, and extricate conclusions that can be made about AK in particular from those that can be made about MMT. When AK is disentangled from standard orthopedic muscle testing, the few studies evaluating unique AK procedures either refute or cannot support the validity of AK procedures as diagnostic tests. The evidence to date does not support the use of MMT for the diagnosis of organic disease or pre/subclinical conditions

Can the feedback of patient assessments, brief training, or their combination, improve the interpersonal skills of primary care physicians? A systematic review

<p>Abstract</p> <p>Background</p> <p>Improving quality of primary care is a key focus of international health policy. Current quality improvement efforts place a large focus on technical, clinical aspects of quality, but a comprehensive approach to quality improvement should also include interpersonal care. Two methods of improving the quality of interpersonal care in primary care have been proposed. One involves the feedback of patient assessments of interpersonal care to physicians, and the other involves brief training and education programmes. This study therefore reviewed the efficacy of (i) feedback of real patient assessments of interpersonal care skills, (ii) brief training focused on the improvement of interpersonal care (iii) interventions combining both (i) and (ii)</p> <p>Methods</p> <p>Systematic review of randomised controlled trials. Three electronic databases were searched (CENTRAL, Medline and Embase) and augmented by searches of the bibliographies of retrieved articles. The quality of studies was appraised and results summarised in narrative form.</p> <p>Results</p> <p>Nine studies were included (two patient based feedback studies and seven brief training studies). Of the two feedback studies, one reported a significant positive effect. Only one training study reported a significant positive effect.</p> <p>Conclusion</p> <p>There is limited evidence concerning the effects of patient based feedback. There is reasonable evidence that brief training as currently delivered is not effective, although the evidence is not definitive, due to the small number of trials and the variation in the training methods and goals. The lack of effectiveness of these methods may reflect a number of issues, such as differences in the effectiveness of the interventions in experienced practitioners and those in training, the lack of theory linking feedback to behaviour change, failure to provide sufficient training or to use a comprehensive range of behaviour change techniques. Further research into both feedback and brief training interventions is required before these interventions are routinely introduced to improve patient satisfaction with interpersonal care in primary care. The interventions to be tested in future research should consider using insights from the wider literature on communication outside primary care, might benefit from a clearer theoretical basis, and should examine the use of combined brief training and feedback.</p