Robustness with skewed data.

<p>Rates of type I errors in UKS test for 3 representative experimental designs (lines) and the 4 skewed distributions shown in <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0039059#pone-0039059-g003" target="_blank">Figure 3B</a> (columns). In each design, the UKS test was applied before and after log-transforming the random datasets. The rates of each design are equal to the percentages of 60000 random datasets with null factor effect that were found significant at the 0.05 threshold by the UKS test. The type I error rates obtained for the same data with Kruskal-Wallis test substituted to Anova are also indicated for the third design. Overall, either log-transformation of skewed data or use of a per-individual nonparametric test guards the UKS test against excessive type I errors.</p

UKS test thresholds and associated p-value limits.

<p>For ten population sizes <i>I</i> from 5 to 30 individuals, the table indicates the Kolmogorov-Smirnov test threshold K_th for type I error rates equal to .05 (column 2) and .01 (column 5). Column 3 and 6 indicate the minimum number <i>n</i>_min of <i>p</i>-values required for the UKS test to be significant. These <i>p</i>-values have to be lower than the limit p_min indicated in columns 4 and 7. Note that the UKS test is significant as soon as <i>n</i>_min + <i>m p</i>-values are below p_min + <i>m</i>/<i>I</i> for any m between 0 and <i>I</i>-<i>n</i>_min. By construction, the limit for <i>I p</i>-values is equal to 1-K_th.</p

Violations of homoscedasticity and normality assumptions in one-way Anova design: compared robustness of RM Anova and UKS test.

<p><i>Panel A:</i> Violation of equal variance assumption. Curves display trial-to-trial errors distributions in the factor levels with the smallest and largest variance for the 4 degrees of heteroscedasticity investigated in simulation studies (see <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0039059#s4" target="_blank">Methods</a>). The numbers under the curves indicate the average percentage of type I errors (false positives) for RM Anovas, individual Anovas and the UKS test procedure, respectively. Numbers above 5% indicate an excess of significant datasets with respect to the tests threshold (0.05). We observe that the UKS test, as the RM Anova, is robust to heterogeneity of variance. <i>Panel B:</i> Violation of normality assumption. Curves display the empirical distributions of trial-to-trial errors drawn from the following 4 distributions: gamma with k = 4; lognormal with μ = 0 and σ = 1/√2; Weibull with k = 1.2 and λ = 0.5; exponential with λ = 0.4 (see <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0039059#s4" target="_blank">Methods</a>). Boxes: Normal probability plots of typical residuals from an Anova applied to skewed data randomly drawn from the above distribution. For the displayed residuals (10 individuals × 3 levels × 10 repetitions with a median coefficient of correlation r), skewness is significant at the .01 threshold when r <0.9942. The numbers under the boxes indicate the across-designs average percentage of type I errors (false positives) for individual Anovas and UKS test applied to raw data or after a logarithmic transformation. Numbers above 5% indicate an excess of significant datasets with respect to the threshold used (0.05). When data is skewed, the UKS test should be used in conjunction with individual nonparametric tests (see text, Part 7), or data should be (log-)transformed.</p

Robustness with violations of heteroscedasticity assumption.

<p>Rates of type I errors in repeated-measures Anovas and UKS test for 3 representative experimental designs (lines) and the same 4 degrees of heteroscedasticity as in <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0039059#pone-0039059-g003" target="_blank">Figure 3A</a> (columns). Rates are averages of designs with 5, 10 and 20 individuals. The rates of each design are equal to the percentages of 60000 random datasets found significant at the 0.05 threshold as the effect of factor was set to zero. Bold values indicate large excess of type I errors. UKS (and RM Anova) are globally robust to violations of heteroscedasticity.</p

Comparison of type II error rates in UKS test and RM Anovas

<p>. Results of a simulation study based on over one billion datasets. Each dataset represents the data of 10 individuals performing 10 trials in each of the 2 levels of a factor. Each data point was obtained by adding to the fixed central value of the level (−1/√2 or +1/√2) two random Gaussian values representing individual idiosyncrasies and trial-to-trial errors (see <a href="http://www.plosone.org/article/info:doi/10.1371/journal.pone.0039059#s4" target="_blank">Methods</a>). <i>Panel A:</i> Median probability (Z-axis) yielded by RM Anovas as a function of the standard deviations of subject-factor interaction (X-axis, rightwards) and average of 10 trial-to-trial errors (Y-axis, leftwards). <i>Panel B:</i> Median probability yielded by the UKS test for the same random data. <i>Panel C:</i> superimposition of the surfaces displayed in panel A and B. Note that in conditions when UKS test is less powerful than ANOVA (larger median p), the difference in power is never dramatic; the converse is not true. <i>Panel D:</i> 2D-isolines of the surfaces in panel C for median probabilities. 001 (red), .01 (orange), .05 (green), .10 (light blue) and .20 (dark blue). Black line: projection of the intersection of the two surfaces; RM Anova is more powerful (smaller median probability) than the UKS test for points leftwards of the black line. Note that scaling the X-axis to the SD of within-level averages of trial-to-trial errors gives a symmetrical aspect to RM Anova surface and projection.</p

Type II errors and reproducibility with heterogeneous experimental effects

<p>. Each panel displays the proportion of significant hypothetical experiments as a function of the difference <i>d</i> between the constant values of experimental effect in 2 (panels A–E) or 3 sub-populations (panel F). The lines show the proportion of significant tests in 10000 hypothetical experiments for 41 values of <i>d</i> from 0 to 8 by .2 steps for RM Anovas (continuous line) and the UKS test at both the .05 (dashed line) and.01 threshold (dotted line). The gray part of lines indicates the 0.211–0.789 range of proportion of significant tests for which the probability that two subsequent experiments yield conflicting outcomes exceeds 1/3. Each experiment consists in 10 individuals performing 8 trials in a baseline condition and in an experimental condition. Trial errors are drawn from a Gaussian distribution with parameters 0 and √8, so that the average of the experimental condition has a Gaussian distribution centered on –<i>d</i>, 0 or +<i>d</i> (Insets) with unitary variance. The proportion and center of the subpopulations varied across studies. In the first study (<i>panel A</i>), the experimental effect was set to 0 for 10% of the population, and to <i>d</i> for the remaining 90%. In the other studies (<i>Panels B–F</i>), the effects and proportions were as follows: [0, 20%; <i>d</i>, 80%]; –<i>d</i>, 10%; <i>d</i>, 90%]; [0, 40%; <i>d</i>, 60%]; –<i>d</i>, 20%; <i>d</i>, 80%]; [–<i>d</i>, 10%; 0; 20%; <i>d</i>, 70%]. For each hypothetical experiment, the 10 individual effects were drawn with replacement from a set of –<i>d</i>, 0 and +<i>d</i> values in the above proportions (for <i>d</i> = 0, the proportion of significant tests is equal to the nominal type I error rate). We conclude that when factor effects vary across individuals as modeled by a mixture of Gaussians, UKS tests yield more reproducible outcomes than RM Anovas and have lower type II errors.</p