Against pragmatism: on efficacy, effectiveness and the real world

Explanatory and pragmatic trials represent ends of a continuum of attitudes about clinical trial design. Recent literature argues that pragmatic trials are more informative about clinical care in the real world. Although there is place for more pragmatic studies to inform clinical practice and health policy decision-making, we are concerned that it is generally under-appreciated that extrapolating the results of broadly inclusive pragmatic trials to the care of real patients may often be as problematic as extrapolating the results of narrowly focused explanatory or efficacy trials. Simplistic interpretation of pragmatic trials runs the risk of driving harmful policies

Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal

Mounting evidence suggests that there is frequently considerable variation in the risk of the outcome of interest in clinical trial populations. These differences in risk will often cause clinically important heterogeneity in treatment effects (HTE) across the trial population, such that the balance between treatment risks and benefits may differ substantially between large identifiable patient subgroups; the "average" benefit observed in the summary result may even be non-representative of the treatment effect for a typical patient in the trial. Conventional subgroup analyses, which examine whether specific patient characteristics modify the effects of treatment, are usually unable to detect even large variations in treatment benefit (and harm) across risk groups because they do not account for the fact that patients have multiple characteristics simultaneously that affect the likelihood of treatment benefit. Based upon recent evidence on optimal statistical approaches to assessing HTE, we propose a framework that prioritizes the analysis and reporting of multivariate risk-based HTE and suggests that other subgroup analyses should be explicitly labeled either as primary subgroup analyses (well-motivated by prior evidence and intended to produce clinically actionable results) or secondary (exploratory) subgroup analyses (performed to inform future research). A standardized and transparent approach to HTE assessment and reporting could substantially improve clinical trial utility and interpretability

Multivariable risk prediction can greatly enhance the statistical power of clinical trial subgroup analysis

BACKGROUND: When subgroup analyses of a positive clinical trial are unrevealing, such findings are commonly used to argue that the treatment's benefits apply to the entire study population; however, such analyses are often limited by poor statistical power. Multivariable risk-stratified analysis has been proposed as an important advance in investigating heterogeneity in treatment benefits, yet no one has conducted a systematic statistical examination of circumstances influencing the relative merits of this approach vs. conventional subgroup analysis. METHODS: Using simulated clinical trials in which the probability of outcomes in individual patients was stochastically determined by the presence of risk factors and the effects of treatment, we examined the relative merits of a conventional vs. a "risk-stratified" subgroup analysis under a variety of circumstances in which there is a small amount of uniformly distributed treatment-related harm. The statistical power to detect treatment-effect heterogeneity was calculated for risk-stratified and conventional subgroup analysis while varying: 1) the number, prevalence and odds ratios of individual risk factors for risk in the absence of treatment, 2) the predictiveness of the multivariable risk model (including the accuracy of its weights), 3) the degree of treatment-related harm, and 5) the average untreated risk of the study population. RESULTS: Conventional subgroup analysis (in which single patient attributes are evaluated "one-at-a-time") had at best moderate statistical power (30% to 45%) to detect variation in a treatment's net relative risk reduction resulting from treatment-related harm, even under optimal circumstances (overall statistical power of the study was good and treatment-effect heterogeneity was evaluated across a major risk factor [OR = 3]). In some instances a multi-variable risk-stratified approach also had low to moderate statistical power (especially when the multivariable risk prediction tool had low discrimination). However, a multivariable risk-stratified approach can have excellent statistical power to detect heterogeneity in net treatment benefit under a wide variety of circumstances, instances under which conventional subgroup analysis has poor statistical power. CONCLUSION: These results suggest that under many likely scenarios, a multivariable risk-stratified approach will have substantially greater statistical power than conventional subgroup analysis for detecting heterogeneity in treatment benefits and safety related to previously unidentified treatment-related harm. Subgroup analyses must always be well-justified and interpreted with care, and conventional subgroup analyses can be useful under some circumstances; however, clinical trial reporting should include a multivariable risk-stratified analysis when an adequate externally-developed risk prediction tool is available

Illusory perceptions of space and time preserve cross-saccadic perceptual continuity

When voluntary saccadic eye movements are made to a silently ticking clock, observers sometimes think that the second hand takes longer than normal to move to its next position. For a short period, the clock appears to have stopped (chronostasis). Here we show that the illusion occurs because the brain extends the percept of the saccadic target backwards in time to just before the onset of the saccade. This occurs every time we move the eyes but it is only perceived when an external time reference alerts us to the phenomenon. The illusion does not seem to depend on the shift of spatial attention that accompanies the saccade. However, if the target is moved unpredictably during the saccade, breaking perception of the target's spatial continuity, then the illusion disappears. We suggest that temporal extension of the target's percept is one of the mechanisms that 'fill in' the perceptual 'gap' during saccadic suppression. The effect is critically linked to perceptual mechanisms that identify a target's spatial stability

Competing risk and heterogeneity of treatment effect in clinical trials

It has been demonstrated that patients enrolled in clinical trials frequently have a large degree of variation in their baseline risk for the outcome of interest. Thus, some have suggested that clinical trial results should routinely be stratified by outcome risk using risk models, since the summary results may otherwise be misleading. However, variation in competing risk is another dimension of risk heterogeneity that may also underlie treatment effect heterogeneity. Understanding the effects of competing risk heterogeneity may be especially important for pragmatic comparative effectiveness trials, which seek to include traditionally excluded patients, such as the elderly or complex patients with multiple comorbidities. Indeed, the observed effect of an intervention is dependent on the ratio of outcome risk to competing risk, and these risks – which may or may not be correlated – may vary considerably in patients enrolled in a trial. Further, the effects of competing risk on treatment effect heterogeneity can be amplified by even a small degree of treatment related harm. Stratification of trial results along both the competing and the outcome risk dimensions may be necessary if pragmatic comparative effectiveness trials are to provide the clinically useful information their advocates intend

Method for evaluating prediction models that apply the results of randomized trials to individual patients

<p>Abstract</p> <p>Introduction</p> <p>The clinical significance of a treatment effect demonstrated in a randomized trial is typically assessed by reference to differences in event rates at the group level. An alternative is to make individualized predictions for each patient based on a prediction model. This approach is growing in popularity, particularly for cancer. Despite its intuitive advantages, it remains plausible that some prediction models may do more harm than good. Here we present a novel method for determining whether predictions from a model should be used to apply the results of a randomized trial to individual patients, as opposed to using group level results.</p> <p>Methods</p> <p>We propose applying the prediction model to a data set from a randomized trial and examining the results of patients for whom the treatment arm recommended by a prediction model is congruent with allocation. These results are compared with the strategy of treating all patients through use of a net benefit function that incorporates both the number of patients treated and the outcome. We examined models developed using data sets regarding adjuvant chemotherapy for colorectal cancer and Dutasteride for benign prostatic hypertrophy.</p> <p>Results</p> <p>For adjuvant chemotherapy, we found that patients who would opt for chemotherapy even for small risk reductions, and, conversely, those who would require a very large risk reduction, would on average be harmed by using a prediction model; those with intermediate preferences would on average benefit by allowing such information to help their decision making. Use of prediction could, at worst, lead to the equivalent of an additional death or recurrence per 143 patients; at best it could lead to the equivalent of a reduction in the number of treatments of 25% without an increase in event rates. In the Dutasteride case, where the average benefit of treatment is more modest, there is a small benefit of prediction modelling, equivalent to a reduction of one event for every 100 patients given an individualized prediction.</p> <p>Conclusion</p> <p>The size of the benefit associated with appropriate clinical implementation of a good prediction model is sufficient to warrant development of further models. However, care is advised in the implementation of prediction modelling, especially for patients who would opt for treatment even if it was of relatively little benefit.</p

Dealing with heterogeneity of treatment effects: is the literature up to the challenge?

<p>Abstract</p> <p>Background</p> <p>Some patients will experience more or less benefit from treatment than the averages reported from clinical trials; such variation in therapeutic outcome is termed heterogeneity of treatment effects (HTE). Identifying HTE is necessary to individualize treatment. The degree to which heterogeneity is sought and analyzed correctly in the general medical literature is unknown. We undertook this literature sample to track the use of HTE analyses over time, examine the appropriateness of the statistical methods used, and explore the predictors of such analyses.</p> <p>Methods</p> <p>Articles were selected through a probability sample of randomized controlled trials (RCTs) published in <it>Annals of Internal Medicine</it>, <it>BMJ</it>, <it>JAMA</it>, <it>The Lancet</it>, and <it>NEJM </it>during odd numbered months of 1994, 1999, and 2004. RCTs were independently reviewed and coded by two abstractors, with adjudication by a third. Studies were classified as reporting: (1) HTE analysis, utilizing a formal test for heterogeneity or treatment-by-covariate interaction, (2) subgroup analysis only, involving no formal test for heterogeneity or interaction; or (3) neither. Chi-square tests and multiple logistic regression were used to identify variables associated with HTE reporting.</p> <p>Results</p> <p>319 studies were included. Ninety-two (29%) reported HTE analysis; another 88 (28%) reported subgroup analysis only, without examining HTE formally. Major covariates examined included individual risk factors associated with prognosis, responsiveness to treatment, or vulnerability to adverse effects of treatment (56%); gender (30%); age (29%); study site or center (29%); and race/ethnicity (7%). Journal of publication and sample size were significant independent predictors of HTE analysis (p < 0.05 and p < 0.001, respectively).</p> <p>Conclusion</p> <p>HTE is frequently ignored or incorrectly analyzed. An iterative process of exploratory analysis followed by confirmatory HTE analysis will generate the data needed to facilitate an individualized approach to evidence-based medicine.</p

Ultrasound screening for asymptomatic carotid stenosis in subjects with calcifications in the area of the carotid arteries on panoramic radiographs: a cross-sectional study

<p>Abstract</p> <p>Background</p> <p>Directed ultrasonic screening for carotid stenosis is cost-effective in populations with > 5% prevalence of the diagnosis. Occasionally, calcifications in the area of the carotid arteries are incidentally detected on odontological panoramic radiographs. We aimed to determine if directed screening for carotid stenosis with ultrasound is indicated in individuals with such calcifications.</p> <p>Methods</p> <p>This was a cross-sectional study. Carotid ultrasound examinations were performed on consecutive persons, with findings of calcifications in the area of the carotid arteries on panoramic radiography that were otherwise eligible for asymptomatic carotid endarterectomy.</p> <p>Results</p> <p>Calcification in the area of the carotid arteries was seen in 176 of 1182 persons undergoing panoramic radiography. Of these, 117 fulfilled the inclusion criterion and were examined with carotid ultrasound. Eight persons (6.8%; 95% CI 2.2-11.5%) had a carotid stenosis - not significant over the 5% pre-specified threshold (p = 0.232, Binomial test). However, there was a significant sex difference (p = 0.008), as all stenoses were found in men. Among men, 12.5% (95%CI 4.2-20.8%) had carotid stenosis - significantly over the 5% pre-specified threshold (p = 0.014, Binomial test).</p> <p>Conclusions</p> <p>The incidental finding of calcification in the area of the carotid arteries on panoramic radiographs should be followed up with carotid screening in men that are otherwise eligible for asymptomatic carotid endarterectomy.</p> <p>Trial Registration</p> <p>The study was registered at <url>http://www.clinicaltrials.gov</url>; <a href="http://www.clinicaltrials.gov/ct2/show/NCT00514644">NCT00514644</a></p

Quadriceps force generation in patients with osteoarthritis of the knee and asymptomatic participants during patellar tendon reflex reactions: an exploratory cross-sectional study

BACKGROUND: It has been postulated that muscle contraction is slower in patients with osteoarthritis of the knee than asymptomatic individuals, a factor that could theoretically impair joint protection mechanisms. This study investigated whether patients with osteoarthritis of the knee took longer than asymptomatic participants to generate force during reflex quadriceps muscle contraction. This was an exploratory study to inform sample size for future studies. METHODS: An exploratory observational cross sectional study was carried out. Two subject groups were tested, asymptomatic participants (n = 17), mean (SD) 56.7 (8.6) years, and patients with osteoarthritis of the knee, diagnosed by an orthopaedic surgeon, (n = 16), age 65.9 (7.8) years. Patellar tendon reflex responses were elicited from participants and measured with a load cell. Force latency, contraction time, and force of the reflex response were determined from digitally stored data. The Mann-Whitney U test was used for the between group comparisons in these variables. Bland and Altman within-subject standard deviation values were calculated to evaluate the measurement error or precision of force latency and contraction time. RESULTS: No significant differences were found between the groups for force latency (p = 0.47), contraction time (p = 0.91), or force (p = 0.72). The two standard deviation measurement error values for force latency were 27.9 ms for asymptomatic participants and 16.4 ms for OA knee patients. For contraction time, these values were 29.3 ms for asymptomatic participants and 28.1 ms for OA knee patients. Post hoc calculations revealed that the study was adequately powered (80%) to detect a difference between the groups of 30 ms in force latency. However it was inadequately powered (59%) to detect this same difference in contraction time, and 28 participants would be required in each group to reach 80% power. CONCLUSION: Patients with osteoarthritis of the knee do not appear to have compromised temporal parameters or magnitude of force generation during patellar tendon reflex reactions when compared to a group of asymptomatic participants. However, these results suggest that larger studies are carried out to investigate this area further

Evaluating treatments in health care: The instability of a one-legged stool

<p>Abstract</p> <p>Background</p> <p>Both scientists and the public routinely refer to randomized controlled trials (RCTs) as being the 'gold standard' of scientific evidence. Although there is no question that placebo-controlled RCTs play a significant role in the evaluation of new pharmaceutical treatments, especially when it is important to rule out placebo effects, they have many inherent limitations which constrain their ability to inform medical decision making. The purpose of this paper is to raise questions about <it>over-reliance </it>on RCTs and to point out an additional perspective for evaluating healthcare evidence, as embodied in the Hill criteria. The arguments presented here are generally relevant to all areas of health care, though mental health applications provide the primary context for this essay.</p> <p>Discussion</p> <p>This article first traces the history of RCTs, and then evaluates five of their major limitations: they often lack external validity, they have the potential for increasing health risk in the general population, they are no less likely to overestimate treatment effects than many other methods, they make a relatively weak contribution to clinical practice, and they are excessively expensive (leading to several additional vulnerabilities in the quality of evidence produced). Next, the nine Hill criteria are presented and discussed as a richer approach to the evaluation of health care treatments. Reliance on these multi-faceted criteria requires more analytical thinking than simply examining RCT data, but will also enhance confidence in the evaluation of novel treatments.</p> <p>Summary</p> <p>Excessive reliance on RCTs tends to stifle funding of other types of research, and publication of other forms of evidence. We call upon our research and clinical colleagues to consider additional methods of evaluating data, such as the Hill criteria. Over-reliance on RCTs is similar to resting all of health care evidence on a one-legged stool.</p