Effects of sampling skewness of the importance-weighted risk estimator on model selection

Importance-weighting is a popular and well-researched technique for dealing with sample selection bias and covariate shift. It has desirable characteristics such as unbiasedness, consistency and low computational complexity. However, weighting can have a detrimental effect on an estimator as well. In this work, we empirically show that the sampling distribution of an importance-weighted estimator can be skewed. For sample selection bias settings, and for small sample sizes, the importance-weighted risk estimator produces overestimates for datasets in the body of the sampling distribution, i.e. the majority of cases, and large underestimates for data sets in the tail of the sampling distribution. These over- and underestimates of the risk lead to suboptimal regularization parameters when used for importance-weighted validation.Comment: Conference paper, 6 pages, 5 figure

Methods for Population Adjustment with Limited Access to Individual Patient Data: A Review and Simulation Study

Population-adjusted indirect comparisons estimate treatment effects when access to individual patient data is limited and there are cross-trial differences in effect modifiers. Popular methods include matching-adjusted indirect comparison (MAIC) and simulated treatment comparison (STC). There is limited formal evaluation of these methods and whether they can be used to accurately compare treatments. Thus, we undertake a comprehensive simulation study to compare standard unadjusted indirect comparisons, MAIC and STC across 162 scenarios. This simulation study assumes that the trials are investigating survival outcomes and measure continuous covariates, with the log hazard ratio as the measure of effect. MAIC yields unbiased treatment effect estimates under no failures of assumptions. The typical usage of STC produces bias because it targets a conditional treatment effect where the target estimand should be a marginal treatment effect. The incompatibility of estimates in the indirect comparison leads to bias as the measure of effect is non-collapsible. Standard indirect comparisons are systematically biased, particularly under stronger covariate imbalance and interaction effects. Standard errors and coverage rates are often valid in MAIC but the robust sandwich variance estimator underestimates variability where effective sample sizes are small. Interval estimates for the standard indirect comparison are too narrow and STC suffers from bias-induced undercoverage. MAIC provides the most accurate estimates and, with lower degrees of covariate overlap, its bias reduction outweighs the loss in effective sample size and precision under no failures of assumptions. An important future objective is the development of an alternative formulation to STC that targets a marginal treatment effect.Comment: 73 pages (34 are supplementary appendices and references), 8 figures, 2 tables. Full article (following Round 4 of minor revisions). arXiv admin note: text overlap with arXiv:2008.0595

Control Variates and Optimal Designs in Metamodeling

At the heart of most modeling issues is a focus on variance reduction. Experimental designs are chosen based on both efficiency and a variety of variance based criteria. In many situations due to cost, time and availability issues it is beneficial to produce metamodels of simulations. Experimental designs for the region of operability are constructed to collect the simulation output required to construct representative metamodels. Independently, the method of control variates is a well established technique often employed to reduce variance in discrete event simulations. This thesis explores the variance reduction benefits that can be obtained by combining optimal experimental designs with control variates in multipopulation simulation experiments when constructing simulation metamodels. A variety of variance measures of effectiveness are used to demonstrate the theoretical benefits obtained by this approach. In addition, a randomly selected data set from within the design region is used to demonstrate the practical application and reduction of predictive variance obtained using this methodology

Applying Bayesian model selection to determine ecological covariates for recruitment and natural mortality in stock assessment

Incorporating ecological covariates into fishery stock assessments may improve estimates, but most covariates are estimated with error. Model selection criteria are often used to identify support for covariates, have some limitations and rely on assumptions that are often violated. For a more rigorous evaluation of ecological covariates, we used four popular selection criteria to identify covariates influencing natural mortality or recruitment in a Bayesian stock assessment of Pacific herring (Clupea pallasii) in Prince William Sound, Alaska. Within this framework, covariates were incorporated either as fixed effects or as latent variables (i.e. covariates have associated error). We found most support for pink salmon increasing natural mortality, which was selected by three of four criteria. There was ambiguous support for other fixed effects on natural mortality (walleye pollock and the North Pacific Gyre Oscillation) and recruitment (hatchery-released juvenile pink salmon and a 1989 regime shift). Generally, similar criteria values among covariates suggest no clear evidence for a consistent effect of any covariate. Models with covariates as latent variables were sensitive to prior specification and may provide potentially very different results. We recommend using multiple criteria and exploring different statistical assumptions about covariates for their use in stock assessment.publishedVersio