Aiming for a representative sample: Simulating random versus purposive strategies for hospital selection

Background A ubiquitous issue in research is that of selecting a representative sample from the study population. While random sampling strategies are the gold standard, in practice, random sampling of participants is not always feasible nor necessarily the optimal choice. In our case, a selection must be made of 12 hospitals (out of 89 Dutch hospitals in total). With this selection of 12 hospitals, it should be possible to estimate blood use in the remaining hospitals as well. In this paper, we evaluate both random and purposive strategies for the case of estimating blood use in Dutch hospitals. Methods Available population-wide data on hospital blood use and number of hospital beds are used to simulate five sampling strategies: (1) select only the largest hospitals, (2) select the largest and the smallest hospitals (‘maximum variation’), (3) select hospitals randomly, (4) select hospitals from as many different geographic regions as possible, (5) select hospitals from only two regions. Simulations of each strategy result in different selections of hospitals, that are each used to estimate blood use in the remaining hospitals. The estimates are compared to the actual population values; the subsequent prediction errors are used to indicate the quality of the sampling strategy. Results The strategy leading to the lowest prediction error in the case study was maximum variation sampling, followed by random, regional variation and two-region sampling, with sampling the largest hospitals resulting in the worst performance. Maximum variation sampling led to a hospital level prediction error of 15 %, whereas random sampling led to a prediction error of 19 % (95 % CI 17 %-26 %). While lowering the sample size reduced the differences between maximum variation and the random strategies, increasing sample size to n = 18 did not change the ranking of the strategies and led to only slightly better predictions. Conclusions The optimal strategy for estimating blood use was maximum variation sampling. When proxy data are available, it is possible to evaluate random and purposive sampling strategies using simulations before the start of the study. The results enable researchers to make a more educated choice of an appropriate sampling strateg

Interval estimation of the overall treatment effect in a meta-analysis of a few small studies with zero events

When a meta-analysis consists of a few small trials that report zero events, accounting for heterogeneity in the (interval) estimation of the overall effect is challenging. Typically, we predefine meta-analytical methods to be employed. In practice, data poses restrictions that lead to deviations from the pre-planned analysis, such as the presence of zero events in at least one study arm. We aim to explore heterogeneity estimators behaviour in estimating the overall effect across different levels of sparsity of events. We performed a simulation study that consists of two evaluations. We considered an overall comparison of estimators unconditional on the number of observed zero cells and an additional one by conditioning on the number of observed zero cells. Estimators that performed modestly robust when (interval) estimating the overall treatment effect across a range of heterogeneity assumptions were the Sidik-Jonkman, Hartung-Makambi and improved Paul-Mandel. The relative performance of estimators did not materially differ between making a predefined or data-driven choice. Our investigations confirmed that heterogeneity in such settings cannot be estimated reliably. Estimators whose performance depends strongly on the presence of heterogeneity should be avoided. The choice of estimator does not need to depend on whether or not zero cells are observed

Interval estimation of the overall treatment effect in a meta-analysis of a few small studies with zero events

When a meta-analysis consists of a few small trials that report zero events, accounting for heterogeneity in the (interval) estimation of the overall effect is challenging. Typically, we predefine meta-analytical methods to be employed. In practice, data poses restrictions that lead to deviations from the pre-planned analysis, such as the presence of zero events in at least one study arm. We aim to explore heterogeneity estimators behaviour in estimating the overall effect across different levels of sparsity of events. We performed a simulation study that consists of two evaluations. We considered an overall comparison of estimators unconditional on the number of observed zero cells and an additional one by conditioning on the number of observed zero cells. Estimators that performed modestly robust when (interval) estimating the overall treatment effect across a range of heterogeneity assumptions were the Sidik-Jonkman, Hartung-Makambi and improved Paul-Mandel. The relative performance of estimators did not materially differ between making a predefined or data-driven choice. Our investigations confirmed that heterogeneity in such settings cannot be estimated reliably. Estimators whose performance depends strongly on the presence of heterogeneity should be avoided. The choice of estimator does not need to depend on whether or not zero cells are observed

Sequential designs with small samples : Evaluation and recommendations for normal responses

Sequential monitoring is a well-known methodology for the design and analysis of clinical trials. Driven by the lower expected sample size, recent guidelines and published research suggest the use of sequential methods for the conduct of clinical trials in rare diseases. However, the vast majority of the developed and most commonly used sequential methods relies on asymptotic assumptions concerning the distribution of the test statistics. It is not uncommon for trials in (very) rare diseases to be conducted with only a few decades of patients and the use of sequential methods that rely on large-sample approximations could inflate the type I error probability. Additionally, the setting of a rare disease could make the traditional paradigm of designing a clinical trial (deciding on the sample size given type I and II errors and anticipated effect size) irrelevant. One could think of the situation where the number of patients available has a maximum and this should be utilized in the most efficient way. In this work, we evaluate the operational characteristics of sequential designs in the setting of very small to moderate sample sizes with normally distributed outcomes and demonstrate the necessity of simple corrections of the critical boundaries. We also suggest a method for deciding on an optimal sequential design given a maximum sample size and some (data driven or based on expert opinion) prior belief on the treatment effect

Level of evidence for promising subgroup findings in an overall non-significant trial

In drug development and drug licensing, it sometimes occurs that a new drug does not demonstrate effectiveness for the full study population, but there appears to be benefit in a relevant, pre-defined subgroup. This raises the question, how strong the evidence from such a subgroup is, and which confirmatory testing strategies are the most appropriate ones. Hence, we considered the type I error and the power of a subgroup result in a trial with non-significant overall results and of suitable replication strategies. In the case of a single trial, the inflation of the overall type I error is substantial and can be up to twice as large, especially in relatively small subgroups. This also increases to the risk of starting a replication trial that should not be done, if such a second trial is not already available. The overall type I error is almost controlled by using an appropriate replication strategy. This confirms the required cautious interpretation of promising subgroups, even in the case that overall trial results were perceived to be close to significance

Subgroup analyses in confirmatory clinical trials : Time to be specific about their purposes

Background: It is well recognized that treatment effects may not be homogeneous across the study population. Subgroup analyses constitute a fundamental step in the assessment of evidence from confirmatory (Phase III) clinical trials, where conclusions for the overall study population might not hold. Subgroup analyses can have different and distinct purposes, requiring specific design and analysis solutions. It is relevant to evaluate methodological developments in subgroup analyses against these purposes to guide health care professionals and regulators as well as to identify gaps in current methodology. Methods: We defined four purposes for subgroup analyses: (1) Investigate the consistency of treatment effects across subgroups of clinical importance, (2) Explore the treatment effect across different subgroups within an overall non-significant trial, (3) Evaluate safety profiles limited to one or a few subgroup(s), (4) Establish efficacy in the targeted subgroup when included in a confirmatory testing strategy of a single trial. We reviewed the methodology in line with this "purpose-based" framework. The review covered papers published between January 2005 and April 2015 and aimed to classify them in none, one or more of the aforementioned purposes. Results: In total 1857 potentially eligible papers were identified. Forty-eight papers were selected and 20 additional relevant papers were identified from their references, leading to 68 papers in total. Nineteen were dedicated to purpose 1, 16 to purpose 4, one to purpose 2 and none to purpose 3. Seven papers were dedicated to more than one purpose, the 25 remaining could not be classified unambiguously. Purposes of the methods were often not specifically indicated, methods for subgroup analysis for safety purposes were almost absent and a multitude of diverse methods were developed for purpose (1). Conclusions: It is important that researchers developing methodology for subgroup analysis explicitly clarify the objectives of their methods in terms that can be understood from a patient's, health care provider's and/or regulator's perspective. A clear operational definition for consistency of treatment effects across subgroups is lacking, but is needed to improve the usability of subgroup analyses in this setting. Finally, methods to particularly explore benefit-risk systematically across subgroups need more research

Estimates of subgroup treatment effects in overall nonsignificant trials : To what extent should we believe in them?

In drug development, it sometimes occurs that a new drug does not demonstrate effectiveness for the full study population but appears to be beneficial in a relevant subgroup. In case the subgroup of interest was not part of a confirmatory testing strategy, the inflation of the overall type I error is substantial and therefore such a subgroup analysis finding can only be seen as exploratory at best. To support such exploratory findings, an appropriate replication of the subgroup finding should be undertaken in a new trial. We should, however, be reasonably confident in the observed treatment effect size to be able to use this estimate in a replication trial in the subpopulation of interest. We were therefore interested in evaluating the bias of the estimate of the subgroup treatment effect, after selection based on significance for the subgroup in an overall “failed” trial. Different scenarios, involving continuous as well as dichotomous outcomes, were investigated via simulation studies. It is shown that the bias associated with subgroup findings in overall nonsignificant clinical trials is on average large and varies substantially across plausible scenarios. This renders the subgroup treatment estimate from the original trial of limited value to design the replication trial. An empirical Bayesian shrinkage method is suggested to minimize this overestimation. The proposed estimator appears to offer either a good or a conservative correction to the observed subgroup treatment effect hence provides a more reliable subgroup treatment effect estimate for adequate planning of future studies

Level of evidence for promising subgroup findings : The case of trends and multiple subgroups

Subgroup analyses are an essential part of fully understanding the complete results from confirmatory clinical trials. However, they come with substantial methodological challenges. In case no statistically significant overall treatment effect is found in a clinical trial, this does not necessarily indicate that no patients will benefit from treatment. Subgroup analyses could be conducted to investigate whether a treatment might still be beneficial for particular subgroups of patients. Assessment of the level of evidence associated with such subgroup findings is primordial as it may form the basis for performing a new clinical trial or even drawing the conclusion that a specific patient group could benefit from a new therapy. Previous research addressed the overall type I error and the power associated with a single subgroup finding for continuous outcomes and suitable replication strategies. The current study aims at investigating two scenarios as part of a nonconfirmatory strategy in a trial with dichotomous outcomes: (a) when a covariate of interest is represented by ordered subgroups, eg, in case of biomarkers, and thus, a trend can be studied that may reflect an underlying mechanism, and (b) when multiple covariates, and thus multiple subgroups, are investigated at the same time. Based on simulation studies, this paper assesses the credibility of subgroup findings in overall nonsignificant trials and provides practical recommendations for evaluating the strength of evidence of subgroup findings in these settings

Towards an appropriate framework to facilitate responsible inclusion of pregnant women in drug development programs

Evidence-based treatment for pregnant women will ultimately require research conducted in the population of pregnant women. Currently, few scholars have addressed the issue of responsible inclusion of pregnant women in drug research. Because of additional risks associated with including pregnant women in drug research and the altered ways in which drugs are processed by the pregnant body, pregnant women cannot be treated as an ordinary subgroup in the various phases of drug development. Instead, responsible inclusion of pregnant women requires careful design and planning of research for pregnant women specifically. Knowledge about these aspects is virtually nonexistent. In this article, we present a practical framework for the responsible inclusion of pregnant women in drug development. We suggest that the framework consists of using a question-based approach with five key questions in combination with three prerequisites which should be addressed when considering inclusion of pregnant women in drug research. The five questions are: A. Can we consider the drug safe (enough) for first exposure in pregnant women and fetuses? B. In which dose range (potentially depending on gestational age) can the drug be considered to remain safe in pregnant women? C. At what dose (regimen, within the range considered safe) can we expect efficacy in pregnant women? D. Can efficacy be confirmed at the target dose, either similar to the initial population or different? E. Can clinical safety be confirmed at a sufficiently acceptable level at the target dose for pregnant women and fetuses, so as to conclude a positive benefit-risk ratio? Combining questions and prerequisites leads to a scheme for appropriate timing of responsible inclusion of pregnant women in drug research. Accordingly, we explore several research design options for including pregnant women in drug trials that are feasible within the framework. Ultimately, the framework may lead to (i) earlier inclusion of pregnant women in drug development, (ii) ensuring that key prerequisites, such as proper dosing, are addressed before more substantial numbers of pregnant women are included in trials, and (iii) optimal use of safety and efficacy data from the initial (nonpregnant) population throughout the drug development process