\u3cem\u3ePowerUp!\u3c/em\u3e: A Tool for Calculating Minimum Detectable Effect Sizes and Minimum Required Sample Sizes for Experimental and Quasi-Experimental Design Studies

This paper complements existing power analysis tools by offering tools to compute minimum detectable effect sizes (MDES) for existing studies and to estimate minimum required sample sizes (MRSS) for studies under design. The tools that accompany this paper support estimates of MDES or MSSR for 21 different study designs that include 14 random assignment designs (6 designs in which individuals are randomly assigned to treatment or control condition and 8 in which clusters of individuals are randomly assigned to condition, with models differing depending on whether the sample was blocked prior to random assignment and by whether the analytic models assume constant, fixed, or random effects across blocks or assignment clusters); and 7 quasi-experimental designs (an interrupted time series design and 6 regression discontinuity designs that vary depending on whether the sample was blocked prior to randomization, whether individuals or clusters of individuals are assigned to treatment or control condition, and whether the analytic models assume fixed or random effects)

Design Considerations in Multisite Randomized Trials Probing Moderated Treatment Effects

Past research has demonstrated that treatment effects frequently vary across sites (e.g., schools) and that such variation can be explained by site-level or individual-level variables (e.g., school size or gender). The purpose of this study is to develop a statistical framework and tools for the effective and efficient design of multisite randomized trials (MRTs) probing moderated treatment effects. The framework considers three core facets of such designs: (a) Level 1 and Level 2 moderators, (b) random and nonrandomly varying slopes (coefficients) of the treatment variable and its interaction terms with the moderators, and (c) binary and continuous moderators. We validate the formulas for calculating statistical power and the minimum detectable effect size difference with simulations, probe its sensitivity to model assumptions, execute the formulas in accessible software, demonstrate an application, and provide suggestions in designing MRTs probing moderated treatment effects

Meaningful Effect Sizes, Intraclass Correlations, and Proportions of Variance Explained by Covariates for Planning Two- and Three-Level Cluster Randomized Trials of Social and Behavioral Outcomes

BACKGROUND: There is a need for greater guidance regarding design parameters and empirical benchmarks for social and behavioral outcomes to inform assumptions in the design and interpretation of cluster randomized trials (CRTs). OBJECTIVES: We calculated the empirical reference values on critical research design parameters associated with statistical power for children's social and behavioral outcomes, including effect sizes, intraclass correlations (ICCs), and proportions of variance explained by a covariate at different levels (R 2). SUBJECTS: Children from kindergarten to Grade 5 in the samples from four large CRTs evaluating the effectiveness of two classroom- and two school-level preventive interventions. MEASURES: Teacher ratings of students' social and behavioral outcomes using the Teacher Observation of Classroom Adaptation-Checklist and the Social Competence Scale-Teacher. RESEARCH DESIGN: Two types of effect size benchmarks were calculated: (1) normative expectations for change and (2) policy-relevant demographic performance gaps. The ICCs and R 2 were calculated using two-level hierarchical linear modeling (HLM), where students are nested within schools, and three-level HLM, where students were nested within classrooms, and classrooms were nested within schools. RESULTS AND CONCLUSIONS: Comprehensive tables of benchmarks and ICC values are provided to inform prevention researchers in interpreting the effect size of interventions and conduct power analyses for designing CRTs of children's social and behavioral outcomes. The discussion also provides a demonstration for how to use the parameter reference values provided in this article to calculate the sample size for two- and three-level CRTs designs

Drawing causal inferences from a longitudinal cluster randomized experiment with crossovers: A study of a distributed leadership program in urban schools

The Distributed Leadership Teacher Training Program (DLT) was implemented to improve student\u27s learning in the United States. However, empirical evidence of the effectiveness of DLT still lacks. This dissertation investigates the effectiveness of DLT using data from the first experimental study of distributed leadership in the United States. The study focuses on the analytic approaches and statistical models to analyze this complex intervention—a longitudinal cluster randomized control trial with cross-overs and unequal probability of treatment assignment. In 2005 the Distributed Leadership Teacher Training Program (DLT) was launched in the School District of Philadelphia funded by the Annenberg Foundation. This multifaceted professional development program for school leadership teams composed of principals and teacher leaders aims to improve student outcomes through developing effective instructional leaders within elementary and high schools under the framework of distributed leadership perspective. The evaluation of the DLT is a longitudinal, cluster randomized experiment. In the 2006-07 school year, 19 eligible elementary schools were recruited and four schools were randomly assigned to the treatment group receiving the intervention. In the 2007-08 school year, seven new elementary schools were recruited. Two out of 22 schools were randomly assigned to the treatment group with twice the probabilities assigned to the original 15 control schools. This dissertation applies 2-level Hierarchical Linear Modeling (HLM) with controlling for the baseline covariates to evaluate the two-year impact of DLT on school leadership teams (Instructional Leadership, and Leadership Data Use), teachers (Teaching Instruction, Interaction with Leader around Instruction, Interaction with Leader around Instruction, Deprivitized Practice/Peer Observation, and Teacher Data Use), and students (school attendance, math, and reading). Except the two-year impact on Deprivitized Practice/Peer Observation , which was significantly positive, no statistical significant results were found in any of the other outcome measures mentioned above at α = .05. However, the two-year impact on math was significantly negative at α = .10. In additional analyses, results based on the experimental design and a second-best non-experimental design are compared to assess the estimate bias of the non-experimental analysis. No significant difference was found from the result comparison between these two approaches. In addition, the issues of statistical model selection and specification in analyzing longitudinal cluster randomized controlled trials were discussed, and the suggestions were made

Drawing causal inferences from a longitudinal cluster randomized experiment with crossovers: A study of a distributed leadership program in urban schools

The Distributed Leadership Teacher Training Program (DLT) was implemented to improve student\u27s learning in the United States. However, empirical evidence of the effectiveness of DLT still lacks. This dissertation investigates the effectiveness of DLT using data from the first experimental study of distributed leadership in the United States. The study focuses on the analytic approaches and statistical models to analyze this complex intervention—a longitudinal cluster randomized control trial with cross-overs and unequal probability of treatment assignment. In 2005 the Distributed Leadership Teacher Training Program (DLT) was launched in the School District of Philadelphia funded by the Annenberg Foundation. This multifaceted professional development program for school leadership teams composed of principals and teacher leaders aims to improve student outcomes through developing effective instructional leaders within elementary and high schools under the framework of distributed leadership perspective. The evaluation of the DLT is a longitudinal, cluster randomized experiment. In the 2006-07 school year, 19 eligible elementary schools were recruited and four schools were randomly assigned to the treatment group receiving the intervention. In the 2007-08 school year, seven new elementary schools were recruited. Two out of 22 schools were randomly assigned to the treatment group with twice the probabilities assigned to the original 15 control schools. This dissertation applies 2-level Hierarchical Linear Modeling (HLM) with controlling for the baseline covariates to evaluate the two-year impact of DLT on school leadership teams (Instructional Leadership, and Leadership Data Use), teachers (Teaching Instruction, Interaction with Leader around Instruction, Interaction with Leader around Instruction, Deprivitized Practice/Peer Observation, and Teacher Data Use), and students (school attendance, math, and reading). Except the two-year impact on Deprivitized Practice/Peer Observation , which was significantly positive, no statistical significant results were found in any of the other outcome measures mentioned above at α = .05. However, the two-year impact on math was significantly negative at α = .10. In additional analyses, results based on the experimental design and a second-best non-experimental design are compared to assess the estimate bias of the non-experimental analysis. No significant difference was found from the result comparison between these two approaches. In addition, the issues of statistical model selection and specification in analyzing longitudinal cluster randomized controlled trials were discussed, and the suggestions were made