Per Family or Familywise Type I Error Control: Eether, Eyether, Neether, Nyther, Let\u27s Call the Whole Thing Off!

Frane (2015) pointed out the difference between per-family and familywise Type I error control and how different multiple comparison procedures control one method but not necessarily the other. He then went on to demonstrate in the context of a two group multivariate design containing different numbers of dependent variables and correlations between variables how the per-family rate inflates beyond the level of significance. In this article I reintroduce other newer better methods of Type I error control. These newer methods provide more power to detect effects than the per-family and familywise techniques of control yet maintain the overall rate of Type I error at a chosen level of significance. In particular, I discuss the False Discovery Rate due to Benjamini and Hochberg (1995) and k-Familywise Type I error control enumerated by Lehmann and Romano (2005), Romano and Shaikh (2006), and Sarkar (2008). I conclude the article by referring readers to articles by Keselman, et al. (2011, 2012) which presented R computer code for determining critical significance levels for these newer methods of Type I error control

A Comparison Of Methods For Longitudinal Analysis With Missing Data

In a longitudinal two-group randomized trials design, also referred to as randomized parallel-groups design or split-plot repeated measures design, the important hypothesis of interest is whether there are differential rates of change over time, that is, whether there is a group by time interaction. Several analytic methods have been presented in the literature for testing this important hypothesis when data are incomplete. We studied these methods for the case in which the missing data pattern is non-monotone. In agreement with earlier work on monotone missing data patterns, our results on bias, sampling variability, Type I error and power support the use of a procedure due to Overall, Ahn, Shivakumar, and Kalburgi (1999) that can easily be implemented with SAS’s PROC MIXE

Assessing Treatment Effects in Randomized Longitudinal Two-Group Designs with Missing Observations

SAS’s PROC MIXED can be problematic when analyzing data from randomized longitudinal two-group designs when observations are missing over time. Overall (1996, 1999) and colleagues found a number of procedures that are effective in controlling the number of false positives (Type I errors) and are yet sensitive (powerful) to detect treatment effects. Two favorable methods incorporate time in study and baseline scores to model the missing data mechanism; one method was a single-stage PROC MIXED ANCOVA solution and the other was a two-stage endpoint analysis using the change scores as dependent scores. Because the twostage approach can lack sensitivity to detect effects for certain missing data mechanisms, in this article we examined variations of the single-stage approach under conditions not considered by Overall et al., in order to assess the generality of the procedure’s positive characteristics. The results indicate when and when not it is beneficial to include a baseline score as a covariate in the model. As well, we provide clarification regarding the merits of adopting an endpoint analysis as compared to the single-stage PROC MIXED procedure

The effects of nonnormality on parametric, nonparametric and model comparison approaches to pairwise comparisons

Researchers in the behavioral sciences are often interested in comparing the means of several treatment conditions on a specific dependent measure. When scores on the dependent measure are not normally distributed, researchers must make important decisions regarding the multiple comparison strategy that is implemented. Although researchers commonly rely on the potential robustness of traditional parametric test statistics (e.g., t and F), these test statistics may not be robust under all nonnormal data conditions. This article compared strategies for performing multiple comparisons with nonnormal data under various data conditions, including simultaneous violations of the assumptions of normality and variance homogeneity. The results confirmed that when variances are unequal, use of the traditional two-sample t test can result in severely biased Type I and/or Type II error rates. However, the use of Welch’s two-sample test statistic with the REGWQ procedure, with either the usual means and variances or with trimmed means and Winsorized variances, resulted in good control of Type I error rates. The Kruskal-Wallis nonparametric statistic provided good Type I error control and power when variances were equal, although Type I error rates became severely inflated when variances were unequal. Furthermore, for researchers interested in eliminating intransitive decisions or comparing potential mean configuration models, a protected model-testing procedure suggested by Dayton provided good overall results.Social Sciences and Humanities Research Counci

Within Groups Multiple Comparisons Based On Robust Measures Of Location

Consider the problem of performing all pair-wise comparisons among J dependent groups based on measures of location associated with the marginal distributions. It is well known that the standard error of the sample mean can be large relative to other estimators when outliers are common. Two general strategies for addressing this problem are to trim a fixed proportion of observations or empirically check for outliers and remove (or down-weight) any that are found. However, simply applying conventional methods for means to the data that remain results in using the wrong standard error. Methods that address this problem have been proposed, but among the situations considered in published studies, no method has been found that gives good control over the probability of a Type I error when sample sizes are small (less than or equal to thirty); the actual probability of a Type I error can drop well below the nominal level. The paper suggests using a slight generalization of a percentile bootstrap method to address this problem

Multivariate Location: Robust Estimators And Inference

The sample mean can have poor efficiency relative to various alternative estimators under arbitrarily small departures from normality. In the multivariate case, (affine equivariant) estimators have been proposed for dealing with this problem, but a comparison of various estimators by Massé and Plante (2003) indicated that the small-sample efficiency of some recently derived methods is rather poor. This article reports that a skipped mean, where outliers are removed via a projection-type outlier detection method, is found to be more satisfactory. The more obvious method for computing a confidence region based on the skipped estimator (using a slight modification of the method in Liu & Singh, 1997) is found to be unsatisfactory except in the bivariate case, at least when the sample size is small. A much more effective method is to use the Bonferroni inequality in conjunction with a standard percentile bootstrap technique applied to the marginal distributions

Robust Confidence Intervals for Effect Size in the Two-Group Case

The probability coverage of intervals involving robust estimates of effect size based on seven procedures was compared for asymmetrically trimming data in an independent two-groups design, and a method that symmetrically trims the data. Four conditions were varied: (a) percentage of trimming, (b) type of nonnormal population distribution, (c) population effect size, and (d) sample size. Results indicated that coverage probabilities were generally well controlled under the conditions of nonnormality. The symmetric trimming method provided excellent probability coverage. Recommendations are provided

Specialized Tests for Detecting Treatment Effects in the Two-Sample Problem

Nonparametric and robust statistics (those using trimmed means and Winsorized variances) were compared for their ability to detect treatment effects in the 2-sample case. In particular, 2 specialized tests, tests designed to be sensitive to treatment effects when the distributions of the data are skewed to the right, were compared with 2 nonspecialized nonparametric (Wilcoxon-Mann-Whitney; Mann & Whitney, 1947; Wilcoxon, 1949) and trimmed (Yuen, 1974) tests for 6 nonnormal distributions that varied according to their measures of. skewness and kurtosis. As expected, the specialized tests provided more power to detect treatment effects, particularly for the nonparametric comparison. However, when distributions were symmetric, the nonspecialized tests were more powerful; therefore, for all the distributions investigated, power differences did not favor the specialized tests. Consequently, the specialized tests are not recommended; researchers would have to know the shapes of the distributions that they work with in order to benefit from specialized tests. In addition, the nonparametric approach resulted in more power than the trimmed-means approach did.Social Sciences and Humanities Research Counci

The Pairwise Multiple Comparison Multiplicity Problem: An Alternative Approach to Familywise and Comparisonwise Type I Error Control

When simultaneously undertaking many tests of significance researchers are faced with the problem of how best to control the probability of committing a Type I error. The familywise approach deals directly with multiplicity problems by setting a level of significance for an entire set (family) of related hypotheses, while the comparison approach ignores the multiplicity issue by setting the rate of error on each individual contrast/test/hypothesis. A new formulation of control presented by Benjamini and Hochberg, their does not provi false discovery rate, de as stringent control as the familywise rate, but concomitant with this relaxing in stringency is an increase in sensitivity to detect effects, compared to familywise control. Type I error and power rates for four relatively powerful and easy-to-compute pairwise multiple comparison procedures were compared to the Benjamni and Hochberg technique for various one-way layouts using test statistics that do not assume variance homogeneity.Social Sciences and Humanities Research Counci