Erfassung des Potenzials zur kognitiven Aktivierung über Unterrichtsmaterialien im Mathematikunterricht

Eine zentrale Voraussetzung der Unterrichtsforschung besteht darin, Unterrichtsmerkmale angemessen zu erfassen. Die Einschätzung des (gefilmten) Unterrichts durch geschulte Rater*innen gilt dabei als Königsweg, ist jedoch mit einem hohen organisatorischen und zeitlichen Aufwand verbunden. Im Rahmen dieses Artikels wird ein neu entwickeltes Erhebungsinstrument für das Unterrichtsqualitätsmerkmal des Potenzials zur kognitiven Aktivierung (PKA) vorgestellt. Das Instrument wurde für Mathematikunterricht zum Thema Quadratische Gleichungen entwickelt, basiert auf der gemeinsamen Auswertung aller Unterrichtsmaterialien einer Stunde durch geschulte Rater*innen und erfasst die von der Lehrperson schriftlich in den Unterricht getragenen Potenziale für kognitive Aktivierung. Die Validität der intendierten Interpretation als Indikator für das schriftlich in den Unterricht eingebrachte PKA einer Unterrichtstunde wird über einen argumentationsbasierten Ansatz untersucht und kann über verschiedene Evidenzen gestützt werden: Beispielsweise zeigt eine D Studie, dass das Instrument bereits von einer einzigen Rater*in zuverlässig erfasst werden kann. Zudem korreliert es substanziell mit einer auf Videoratings basierenden Messung des PKA. (DIPF/Orig.)A central challenge of instructional research is to measure teaching characteristics appropriately. The assessment of (filmed) instruction by trained raters is regarded as the best way to achieve this, but is associated with high organizational and time costs. In this paper, we present a newly developed instrument for assessing the teaching quality characteristic potential for cognitive activation (PCA). The instrument was developed for mathematics lessons on the topic of quadratic equations. It is based on the joint evaluation of all instructional artifacts of a lesson (lesson plans, student assignments, assessments, etc.) by trained raters and measures the potential for cognitive activation brought into the classroom by the teacher in writing. The validity of the intended interpretation as an indicator for the written PCA of a lesson is examined by an argument-based approach and can be supported by various pieces of evidence: For example, a decision study shows that the instrument can already be reliably assessed by a single rater. Furthermore, it correlates substantially with a measurement of the PCA based on video ratings. (DIPF/Orig.

Learning From Summer: Effects of Voluntary Summer Learning Programs on Low-Income Urban Youth

The largest-ever study of summer learning finds that students with high attendance in free, five to six-week, voluntary summer learning programs experienced educationally meaningful benefits in math and reading.The findings are important because children from low-income families lose ground in learning over the summer compared to their more affluent peers. Voluntary, district-run summer programs could help shrink this gap and have the potential to reach more students than traditional summer school or smaller-scale programs run by outside organizations. Yet until now little has been known about the impact of these programs and how they can succeed. Wallace's $50 million National Summer Learning Project seeks to help provide answers.Since 2011, five urban school districts and their partners, the RAND Corporation and Wallace have been working together to find out whether and how voluntary-attendance summer learning programs combining academics and enrichment can help students succeed in school.Starting in 2013, RAND conducted a randomized controlled trial (RCT) in five districts—Boston; Dallas; Duval County, Florida; Pittsburgh; and Rochester—to evaluate educational outcomes, focusing on children who were in 3rd grade in spring of that year. The 5,600 students who applied to summer programs were randomly assigned to one of two groups—those selected to take part in the programs for two summers (the treatment group) and those not selected (the control group). The study analyzed outcomes for 3,192 students offered access to the programs.Researchers found that those who attended a five-to-six-week summer program for 20 or more days in 2013 did better on state math tests than similar students in the control group. This advantage was statistically significant and lasted through the following school year. The results are even more striking for high attenders in 2014: They outperformed control group students in both math and English Language Arts (ELA), on fall tests and later, in the spring. The advantage after the second summer was equivalent to 20-25 percent of a year's learning in math and ELA.These findings are correlational but controlled for prior achievement and demographics, giving researchers confidence that the benefits are likely due to the programs and meeting the requirements for promising evidence under the Every Student Succeeds Act.High-attending students were also rated by teachers as having stronger social and emotional competencies than the control group students; however, researchers have less confidence that this was due to the programs, given the lack of prior data on these competencies.About 60 percent of students attending at least one day met the 20-day threshold that was defined as high attendance.Separately, the study also examined the impact of the programs on all students who were offered access, whether or not they actually attended. Because many students did not attend at a high level, and some didn't attend at all, the average benefits for all of these students were smaller and not statistically significant, with the exception of a modest but educationally meaningful boost in math scores in the fall after the first summer equivalent to 15 percent of a year's learning. These findings are causal, meaning that researchers are confident that they were due to the programs, and meet the standard of strong evidence under the Every Student Succeeds Act.For students to experience lasting benefits from attending summer programs, the report recommends that districts: run programs for at least five weeks; promote high attendance; include sufficient instructional time and protect it; invest in instructional quality; and factor in attendance and likely no-show rates when staffing the programs in order to lower per-student costs

Multilevel Factor Analysis and Student Ratings of Instructional Practice

Student surveys of classroom climate can provide teachers and researchers with valuable information about instruction. These surveys are becoming a critical component in policy efforts to assess and improve teaching. Advocates note that students are natural observers of their classroom environments, have extensive and rich knowledge of their teachers, and that student ratings of teacher practice can be predictive of important outcomes, such as student academic and socio-emotional development. Inferences about a teacher's instructional practice are often based on aggregated student survey responses, and a key step in assessing the appropriate uses of the information collected from student surveys is to understand the dimensions of classroom climate or instructional practice that are discernible when looking at student responses aggregated by classroom. This dissertation proposes a new approach for exploring the dimensionality of aggregated student ratings. This approach also has the potential to provide validity evidence supporting the use of student surveys as measures of instructional practice in both formative and summative evaluation. Specifically, this dissertation applies a non-parametric cluster bootstrap technique to a multilevel covariance structure analysis framework that allows researchers to evaluate and investigate psychometric models when data is collected from students and teachers are the objects of measurement. This approach is extended to applications where teachers are clustered within schools. The cluster bootstrap technique is demonstrated on a realistic dataset to illustrate how the methods may be used to investigate the dimensions of teacher professional practice that are discernible on a state-wide student survey of instructional practice. The results of this dissertation demonstrate that the proposed cluster bootstrap technique can be used in conjunction with maximum likelihood estimation to yield accurate parameter estimates, and that for sufficiently large sample sizes, test statistics and standard errors based on the cluster bootstrap technique will yield valid inferences about the psychometric properties of aggregated survey responses

Multilevel Factor Analysis by Model Segregation: Comparing the Performance of Maximum Likelihood and Robust Test Statistics

Survey measures of classroom climate and instructional practice have become central to policy efforts that assess school and teacher quality. Measures of classroom climate are often formed by aggregating individual survey responses. This has sparked a wide interest in using multilevel factor analysis to test hypotheses about the psychometric properties of classroom climate variables. One approach to multilevel factor analysis is conducted in two steps. First, the total covariance matrix is partitioned into separate between-group and within-group covariance matrices. Second, conventional factor analysis is used to test models separately. This study shows that when using this approach, rescaled and residual-based test statistics provide better inferences about the between group-level measurement structure than Maximum Likelihood test statistics even when the number of groups is large and there is no excess kurtosis in the observed variables. This study presents an empirical example and a simulation study to demonstrate how item intraclass correlations and within group sample sizes influence test statistic performance