Design Considerations in Multisite Randomized Trials Probing Moderated Treatment Effects

Past research has demonstrated that treatment effects frequently vary across sites (e.g., schools) and that such variation can be explained by site-level or individual-level variables (e.g., school size or gender). The purpose of this study is to develop a statistical framework and tools for the effective and efficient design of multisite randomized trials (MRTs) probing moderated treatment effects. The framework considers three core facets of such designs: (a) Level 1 and Level 2 moderators, (b) random and nonrandomly varying slopes (coefficients) of the treatment variable and its interaction terms with the moderators, and (c) binary and continuous moderators. We validate the formulas for calculating statistical power and the minimum detectable effect size difference with simulations, probe its sensitivity to model assumptions, execute the formulas in accessible software, demonstrate an application, and provide suggestions in designing MRTs probing moderated treatment effects

Predicting Second and Third Graders' Reading Comprehension Gains: Observing Students' and Classmates Talk during Literacy Instruction using COLT.

This paper introduces a new observation system that is designed to investigate students' and teachers' talk during literacy instruction, Creating Opportunities to Learn from Text (COLT). Using video-recorded observations of 2nd-3rd grade literacy instruction (N=51 classrooms, 337 students, 151 observations), we found that nine types of student talk ranged from using non-verbal gestures to generating new ideas. The more a student talked, the greater were his/her reading comprehension (RC) gains. Classmate talk also predicted RC outcomes (total effect size=0.27). We found that 11 types of teacher talk ranged from asking simple questions to encouraging students' thinking and reasoning. Teacher talk predicted student talk but did not predict students' RC gains directly. Findings highlight the importance of each student's discourse during literacy instruction, how classmates' talk contributes to the learning environments that each student experiences, and how this affects RC gains, with implications for improving the effectiveness of literacy instruction

Complexidade e escala na investigação da eficácia do ensino: reflexões do estudo MET

Researchers and policymakers in the US and beyond increasingly seek to identify teaching qualities that are associated with academic achievement gains for K-12 students through effectiveness studies. Yet teaching quality varies with academic content and social contexts, involves multiple participants, and requires a range of skills, knowledge, and dispositions. In this essay, we address the inescapable tension between complexity and scale in research on teaching effectiveness. We provide five recommendations to study designers and analysts to manage this tension to enhance effectiveness research, drawing on our recent experiences as the first external analysts of the Measures of Effective Teaching (MET) study. Our recommendations address conceptual framing, the measurement of teaching (e.g., observation protocols, student surveys), sampling, classroom videoing, and the use and interpretation of value-added models.Investigadores y legisladores en los Estados Unidos y en otros países buscan identificar las cualidades de la enseñanza que se asocian con incrementos de desempeño académico para alumnos de primaria y secundaria a través de estudios de eficacia. Sin embargo, la calidad de la enseñanza varía según el contenido académico y los contextos sociales, involucra a múltiples participantes y requiere una variedad de habilidades, conocimientos y disposiciones. En este ensayo, abordamos la ineludible tensión entre la complejidad y la escala en la investigación sobre la eficacia de la enseñanza. Proveemos cinco recomendaciones a los diseñadores de estudios y analistas para manejar esta tensión y mejorar la investigación de eficacia, aprovechando nuestras experiencias recientes como los primeros analistas externos del estudio Measures of Effective Teaching (MET). Nuestras recomendaciones abordan el marco conceptual, la medición de la enseñanza (por ej., protocolos de observación, encuestas de estudiantes), el muestreo, el video en el aula y el uso e interpretación de modelos de valor agregado.Pesquisadores e legisladores nos Estados Unidos e em outros países buscam identificar as qualidades de ensino associadas ao aumento do desempenho acadêmico de alunos do ensino fundamental e médio por meio de estudos de eficácia. No entanto, a qualidade do ensino varia de acordo com o conteúdo acadêmico e os contextos sociais, envolve múltiplos participantes e requer uma variedade de habilidades, conhecimentos e disposições. Neste ensaio, abordamos a tensão inescapável entre complexidade e escala na pesquisa sobre a eficácia do ensino. Fornecemos cinco recomendações para projetistas e analistas de estudo para gerenciar essa tensão e melhorar a pesquisa sobre eficácia, alavancando nossas experiências recentes como os primeiros analistas externos do estudo Measures of Effective Teaching (MET). Nossas recomendações abordam a estrutura conceitual, a medição do ensino (por exemplo, protocolos de observação, pesquisas com estudantes), amostragem, vídeo em sala de aula e o uso e interpretação de modelos de valor agregado

Improving and Assessing Propensity Score Based Causal Inferences in Multilevel and Nonlinear Settings.

Recent calls for accountability have focused on scientifically based research that isolates causal mechanisms to inform both the policies and practices of education. A major challenge in aligning educational research with such standards has been to develop methods that can address the interdependency and multilevel structure of teaching and learning and approximate randomized experiments using observational data. In this dissertation, I carried out three studies that centered on improving causal inferences drawn from observational studies in common educational settings. In the first study, I developed several models for estimating multilevel propensity scores (PSs) and examined their effectiveness for causal inference. The results suggested consistent gains from multilevel PSs that allow differential influence of the group on its individuals. The results further suggested that covariate selection in multilevel PSs can play a large role, both relative to model type and in an absolute sense. The second study then developed a method to construct PSs in an effective and efficient manner using two pivotal relationships. The method made use of each covariate’s relationship with the treatment and commonly available outcome proxies (e.g. pretest measures) to construct PSs that minimizes the mean-square error (MSE) of the treatment effect estimator. The results of the study suggested that an effective and efficient approach to constructing the PS might be to include those covariates whose relationship with the outcome is at least half the magnitude of the respective relationship with the treatment. In the final study, I develop an index that assesses the sensitivity of inferences in binomial regression models by extending the impact threshold of a confounding variable framework (Frank, 2000). Each of these methods is then applied to observational data to demonstrate how these methods can advance the quality and robustness of causal inferences in educational research.Ph.D.Education StudiesUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/63716/1/bkelcey_1.pd

Predicting Second and Third Graders' Reading Comprehension Gains: Observing Students' and Classmates Talk during Literacy Instruction using COLT.

This paper introduces a new observation system that is designed to investigate students' and teachers' talk during literacy instruction, Creating Opportunities to Learn from Text (COLT). Using video-recorded observations of 2nd-3rd grade literacy instruction (N=51 classrooms, 337 students, 151 observations), we found that nine types of student talk ranged from using non-verbal gestures to generating new ideas. The more a student talked, the greater were his/her reading comprehension (RC) gains. Classmate talk also predicted RC outcomes (total effect size=0.27). We found that 11 types of teacher talk ranged from asking simple questions to encouraging students' thinking and reasoning. Teacher talk predicted student talk but did not predict students' RC gains directly. Findings highlight the importance of each student's discourse during literacy instruction, how classmates' talk contributes to the learning environments that each student experiences, and how this affects RC gains, with implications for improving the effectiveness of literacy instruction