Development and evaluation of an instrument for the critical appraisal of randomized controlled trials of natural products

<p>Abstract</p> <p>Background</p> <p>The efficacy of natural products (NPs) is being evaluated using randomized controlled trials (RCTs) with increasing frequency, yet a search of the literature did not identify a widely accepted critical appraisal instrument developed specifically for use with NPs. The purpose of this project was to develop and evaluate a critical appraisal instrument that is sufficiently rigorous to be used in evaluating RCTs of conventional medicines, and also has a section specific for use with single entity NPs, including herbs and natural sourced chemicals.</p> <p>Methods</p> <p>Three phases of the project included: 1) using experts and a Delphi process to reach consensus on a list of items essential in describing the identity of an NP; 2) compiling a list of non-NP items important for evaluating the quality of an RCT using systematic review methodology to identify published instruments and then compiling item categories that were part of a validated instrument and/or had empirical evidence to support their inclusion and 3) conducting a field test to compare the new instrument to a published instrument for usefulness in evaluating the quality of 3 RCTs of a NP and in applying results to practice.</p> <p>Results</p> <p>Two Delphi rounds resulted in a list of 15 items essential in describing NPs. Seventeen item categories fitting inclusion criteria were identified from published instruments for conventional medicines. The new assessment instrument was assembled based on content of the two lists and the addition of a Reviewer's Conclusion section. The field test of the new instrument showed good criterion validity. Participants found it useful in translating evidence from RCTs to practice.</p> <p>Conclusion</p> <p>A new instrument for the critical appraisal of RCTs of NPs was developed and tested. The instrument is distinct from other available assessment instruments for RCTs of NPs in its systematic development and validation. The instrument is ready to be used by pharmacy students, health care practitioners and academics and will continue to be refined as required.</p

Reporting and Concordance of Methodologic Criteria Between Abstracts and Articles in Diagnostic Test Studies

OBJECTIVE: To evaluate the quality and concordance of methodologic criteria in abstracts versus articles regarding the diagnosis of trichomoniasis. STUDY DESIGN: Survey of published literature. DATA SOURCES: Studies indexed in medline(1976–1998). STUDY SELECTION: Studies that used culture as the gold or reference standard. DATA EXTRACTION: Data from abstract and articles were independently abstracted using 4 methodologic criteria: (1) prospective evaluation of consecutive patients; (2) test results did not influence the decision to do gold standard; (3) independent and blind comparison with gold standard; and (4) broad spectrum of patients used. The total number of criteria met for each report was calculated to create a quality score (0–4). MEASUREMENTS AND MAIN RESULTS: None of the 33 abstracts or full articles reported all 4 criteria. Three criteria were reported in none of the abstracts and in 18% of articles (95% confidence interval [95% CI] 8.6% to 34%). Two criteria were reported in 18% of abstracts (95% CI, 8.6% to 34%) and 42% of articles (95% CI, 27% to 59%). One criterion was reported in 42% of abstracts (95% CI, 27% to 59%) and 27% of articles (95% CI, 15% to 44%). No criteria were reported in 13 (39%) of 33 abstracts (95% CI, 25% to 56%) and 4 (12%) of 33 articles (95% CI, 4.8% to 27%). The agreement of the criteria between the abstract and the article was poor (κ−0.09; 95% CI, −0.18 to 0) to moderate (κ 0.53; 95% CI, 0.22 to 0.83). CONCLUSIONS: Information on methods basic to study validity is often absent from both abstract and paper. The concordance of such criteria between the abstract and article needs to improve