No short-cut in assessing trial quality: a case study

Assessing the quality of included trials is a central part of a systematic review. Many check-list type of instruments for doing this exist. Using a trial of antibiotic treatment for acute otitis media, Burke et al., BMJ, 1991, as the case study, this paper illustrates some limitations of the check-list approach to trial quality assessment. The general verdict from the check list type evaluations in nine relevant systematic reviews was that Burke et al. (1991) is a good quality trial. All relevant meta-analyses extensively used its data to formulate therapeutic evidence. My comprehensive evaluation, on the other hand, brought to the surface a series of serious problems in the design, conduct, analysis and report of this trial that were missed by the earlier evaluations. A check-list or instrument based approach, if used as a short-cut, may at times rate deeply flawed trials as good quality trials. Check lists are crucial but they need to be augmented with an in-depth review, and where possible, a scrutiny of the protocol, trial records, and original data. The extent and severity of the problems I uncovered for this particular trial warrant an independent audit before it is included in a systematic review

Relationship between time-integrated disease activity estimated by DAS28-CRP and radiographic progression of anatomical damage in patients with early rheumatoid arthritis

<p>Abstract</p> <p>Background</p> <p>The main aim of the study was to investigate the relationship between persistent disease activity and radiographic progression of joint damage in early rheumatoid arthritis (ERA).</p> <p>Methods</p> <p>Forty-eight patients with active ERA was assessed every 3 months for disease activity for 3 years. Radiographic damage was measured by the Sharp/van der Heijde method (SHS). The cumulative inflammatory burden was estimated by the time-integrated values (area under the curve-AUC) of Disease Activity Score 28 joint based on C-reactive protein (DAS28-CRP) in rapid progressors versus non-progressors. Bland and Altman's 95% limits of agreement method were used to estimate the smallest detectable difference (SDD) of radiographic progression. The relationship between clinical and laboratory predictors of radiographic progression and their interactions with time was analysed by logistic regression model.</p> <p>Results</p> <p>After 3-years of follow-up, radiographic progression was observed in 54.2% (95%CI: 39.8% to 67.5%) of patients and SDD was 9.5 for total SHS. The percentage of patients with erosive disease increased from 33.3% at baseline to 76% at 36 months. The total SHS of the progressors worsened from a median (interquartile range) of 18.5 (15-20) at baseline to 38.5 (34-42) after 3 years (p < 0.0001) whereas non-progressors worsened from a median of 14.5 (13-20) at baseline to 22.5 (20-30) after 3 years (p < 0.001). In the regression model, time-integrated values of DAS28-CRP and anti-CCP positivity have the highest positive predictive value for progression (both at level of p < 0.0001). Radiographic progression was also predicted by a positive IgM-RF (p0.0009), and a high baseline joint damage (p = 0.0044).</p> <p>Conclusions</p> <p>These data indicate that the level of disease activity, as measured by time-integrated DAS28-CRP, anti-CCP and IgM-RF positivity and a high baseline joint damage, affects subsequent progression of radiographic damage in ERA.</p

LDL-cholesterol differences predicted survival benefit in statin trials by the surrogate threshold effect (STE)

Objective: We describe a new statistical method called the surrogate threshold effect (STE) that estimates the threshold level of a surrogate needed in a clinical trial to predict a benefit in the target clinical outcome. In this article, we apply this method to the LDL-cholesterol biomarker surrogate and survival benefit-target outcome in statin trials. Study Design and Setting: We identified randomized trials comparing statin treatment to placebo treatment or no treatment and reporting all-cause and cardiovascular mortality. Trials with fewer than five all-cause deaths in at least one arm were excluded. Multiple regression modeled the reduction in all-cause and cardiovascular mortality as a function of LDL-cholesterol difference. The 95% confidence and 95% prediction bands were calculated and graphed to determine the minimum LDL-cholesterol difference (the surrogate threshold) below which there would be no predicted survival benefit. Results: In 16 qualifying trials, regression analysis yielded an all-cause mortality model whose prediction bands demonstrated no overall survival gain with LDL-cholesterol difference values below 1.5 mmol/L. The cardiovascular mortality model yielded prediction bands that demonstrated no cardiovascular survival benefit with LDL-cholesterol difference values below 1.4 mmol/L. Conclusions: In a multitrial setting, the STE approach is a promising yet straightforward statistical method for evaluating the surrogate validity of biomarkers

Outcome variables for osteoarthritis clinical trials: the OMERACT-OARSI set of responder criteria

Improvement in analysis and reporting results of osteoarthritis (OA) clinical trials has been recently obtained because of harmonization and standardization of the selection of outcome variables (OMERACT 3 and OARSI). Moreover, OARSI has recently proposed the OARSI responder criteria. This composite index permits presentation of results of symptom modifying clinical trials in OA based on individual patient responses (responder yes/no). The 2 organizations (OMERACT and OARSI) established. a task force aimed at evaluating: (1) the variability of observed placebo and active treatment effects using the OARSI responder criteria; and (2) the possibility of proposing a simplified set of criteria. The conclusions of the task force were presented and discussed during the OMERACT 6 conference, where a simplified set of responder criteria (OMERACT-OARSI set of criteria) was proposed

Standardized assessment of adverse events in rheumatology clinical trials: Summary of the OMERACT 7 Drug Safety Module update

A presentation, demonstration, and discussion of recently developed adverse event instruments were the topics for the OMERACT 7 Drug Safety Module. The module began with a plenary introducing the needs and challenges of adverse event ascertainment. It was followed by a review of module work from previous OMERACT meetings on a prototype coding instrument (Rheumatology Common Toxicity Criteria), then a brief description of the process behind the recently developed patient self-report and investigator report adverse event instruments. These current instruments are designed for use in controlled trials although they could be used in other settings. The instruments rely primarily on patient self-reporting using a checklist, which the investigator then folds into a parallel structured but more medically sophisticated instrument. In pilot testing, this innovative dual-use system has shown reliability and acceptability, while preserving validity. A "stakeholder panel" of representatives from 8 sectors followed - patient, nurse investigator, regulator, clinician scientist, industry, OMERACT, global public health/WHO, and Cochrane Collaboration - for their perspectives on the needs, challenges, and potential ways forward for adverse event ascertainment and reporting in clinical trials. At the breakout session small focus groups participated in hands-on interactive testing of one of 3 versions of the instruments, which differ in degree of comprehensiveness. Each focus group had a participatory patient with rheumatoid arthritis. At a second plenary there was group feedback by rapporteurs and presentation of results from pilot studies of iterative testing of validity, reliability, and feasibility of the instruments. During plenary discussion a frequent suggestion for improvement was to refine the process so that event ascertainment could be done entirely using the patient instrument with minimal input from the investigator at the visit, if patient-investigator agreement was high. Most found the patient checklist attractive, particularly if the patient instrument was shown to be reliable and valid. Finally, a future research agenda was discussed