Supplemental Material for Newman et al., 2018

Comparison of Event analysis with RSEM to Event Analysis with eXpress and these approaches to transcript estimation with iRecko

Variable selection in omics data: A practical evaluation of small sample sizes

<div><p>In omics experiments, variable selection involves a large number of metabolites/ genes and a small number of samples (the <i>n</i> < <i>p</i> problem). The ultimate goal is often the identification of one, or a few features that are different among conditions- a biomarker. Complicating biomarker identification, the <i>p</i> variables often contain a correlation structure due to the biology of the experiment making identifying causal compounds from correlated compounds difficult. Additionally, there may be elements in the experimental design (blocks, batches) that introduce structure in the data. While this problem has been discussed in the literature and various strategies proposed, the over fitting problems concomitant with such approaches are rarely acknowledged. Instead of viewing a single omics experiment as a definitive test for a biomarker, an unrealistic analytical goal, we propose to view such studies as screening studies where the goal of the study is to reduce the number of features present in the second round of testing, and to limit the Type II error. Using this perspective, the performance of LASSO, ridge regression and Elastic Net was compared with the performance of an ANOVA via a simulation study and two real data comparisons. Interestingly, a dramatic increase in the number of features had no effect on Type I error for the ANOVA approach. ANOVA, even without multiple test correction, has a low false positive rates in the scenarios tested. The Elastic Net has an inflated Type I error (from 10 to 50%) for small numbers of features which increases with sample size. The Type II error rate for the ANOVA is comparable or lower than that for the Elastic Net leading us to conclude that an ANOVA is an effective analytical tool for the initial screening of features in omics experiments.</p></div

Summary of the simulation scenarios.

<p>For each scenario 1000 datasets were simulated. All together there were 20 scenarios considered.</p

Visualization of power and Type I error estimates comparison for <i>p</i> = 205 features, correlation <i>ρ</i> = 0.4, and all sample sizes.

<p>Each row of the plots corresponds to a feature selection method. ANOVA FDR adjustment cutoff is 0.2. The value of the sample size (<i>n</i>) is displayed on the <i>x</i>-axis in all plots. The estimates of power and Type I error are provided on the <i>y</i>-axis. In the left column power estimates are provided based on the four different features for each of the effect sizes (Δ₁ = 0.2 is the red line, Δ₂ = 0.5 is the blue line, and Δ₃ = 0.8 is the green line). In the right column Type I error estimates are provided (beige lines) based on the random noise features together with a 0.05 threshold plotted as a purple dashed line. In the middle column the proportions of non-different detected features within each block correlated to different ones for each of the blocks and corresponding effect sizes (Δ₁ = 0.2 is the red line, Δ₂ = 0.5 is the blue line, and Δ₃ = 0.8 is the green line) are displayed.</p

Variable selection in omics data: A practical evaluation of small sample sizes - Fig 1

<p>Visualization of power (left column) and Type I error (right column) estimates comparison between <i>p</i> = 205 (solid lines) and <i>p</i> = 2050 (dashed line) features for <i>ρ</i> = 0.4 and sample sizes <i>n</i> = 10 (top row), <i>n</i> = 50 (middle row), and <i>n</i> = 100 (bottom row). The value of the penalty split parameter <i>α</i> is plotted on the <i>x</i>-axis. Type I error and power estimates are plotted on <i>y</i>-axis for the values of <i>α</i> in the range of [0; 1] with 0.1 increments. In the left column power estimates are provided based on the four different features for each of the effect sizes (Δ₁ = 0.2 is the red line, Δ₂ = 0.5 is the blue line, and Δ₃ = 0.8 is the green line). In the right column Type I error estimates are provided (beige lines) based on the random noise features together with a 0.05 threshold plotted as a purple dashed line. The vertical dashed line in the right column plots corresponds to penalty split value <i>α</i> = 0.5. The value of <i>α</i> = 0 corresponds to ridge regression and <i>α</i> = 1 corresponds to LASSO.</p

Variable selection in omics data: A practical evaluation of small sample sizes - Fig 2

<p>Visualization of power (left column) and Type I error (right column) estimates. Comparison between <i>p</i> = 205 (solid lines) and <i>p</i> = 2050 (dashed line) features for <i>ρ</i> = 0.8 and sample sizes <i>n</i> = 10 (top row), <i>n</i> = 50 (middle row), and <i>n</i> = 100 (bottom row). The value of the penalty split parameter <i>α</i> is plotted on the <i>x</i>-axis. Type I error and power estimates are plotted on <i>y</i>-axis for the values of <i>α</i> in the range of [0; 1] with 0.1 increments. In the left column power estimates are provided based on the four different features for each of the effect sizes (Δ₁ = 0.2 is the red line, Δ₂ = 0.5 is the blue line, and Δ₃ = 0.8 is the green line). In the right column Type I error estimates are provided (beige lines) based on the random noise features together with a 0.05 threshold plotted as a purple dashed line. The vertical dashed line in the right column plots corresponds to penalty split value <i>α</i> = 0.5. The value of <i>α</i> = 0 corresponds to ridge regression and <i>α</i> = 1 corresponds to LASSO.</p

Gene-environment interactions.

<p>^a major/minor allele</p><p>^b odds ratio</p><p>^c adjusted for age and gender</p><p>^d ≥ 17 pack years (median)</p><p>^e lifetime exposure to pesticide >26 days.</p><p>Gene-environment interactions.</p

Study demographics.

<p>^a Odds ratio (95% CI) calculated using binary logistic regression</p><p>^b Wilcoxon rank sum test</p><p>^c pack years equivalent to packs per day multiplied by years smoked</p><p>^d pack years dichotomised according to median <17 & ≥ 17 pack years (zero pack years removed)</p><p>^e Pesticide exposure is equivalent to exposure of herbicides, pesticides, or fungicides at least once weekly for a period of six months.</p><p>Study demographics.</p

Haplotype construction map (VO1-VO7 & T1–T7) and associated AAO effects.

<p>NS = not significant (>0.05).</p

Genotyping associations of SNPs and Parkinson’s disease.

<p>^a Major/minor allele</p><p>^b Odds ratio</p><p>^c adjusted for age and gender</p><p>^d Bonferroni corrected <i>P</i><0.05.</p><p>Genotyping associations of SNPs and Parkinson’s disease.</p