Evaluating Teacher Effectiveness Using Classroom Observations: A Rasch Analysis of the Rater Effects of Principals

In most U.S. schools, teachers are evaluated using observation of teaching practice (OTP). This study investigates rater effects on OTP ratings among 421 principals in an authentic teacher evaluation system. Many-facet Rasch analysis (MFR) using a block of shared ratings revealed that principals generally (a) differentiated between more and less effective teachers, (b) rated their teachers with leniency (i.e., overused higher rating categories), and (c) differentiated between teaching practices (e.g., Cognitive Engagement vs. Classroom Management) with minimal halo effect. Individual principals varied significantly in degree of leniency, and approximately 12% of principals exhibited severe rater bias. Implications for use of OTP ratings for evaluating teachers’ effectiveness are discussed. Strengths and limitations of MFR to analyze rater effects in OTP are also discussed

Child and Adolescent development in your classroom

xxviii, 676 hal.; ilus.; 28 cm

Principals may inflate teacher evaluation scores to achieve important goals

A concerning attribute of teacher evaluations across countries is the systemic leniency of principals during classroom observations. However, little is known about the motivations behind this phenomenon. The purpose of this study is to explore the motivating factors behind principals’ leniency in an authentic teacher evaluation system. In this study, we apply an explanatory-sequential mixed-method design. Using focus groups (qualitative strand; n = 15 principals) and a state-wide survey (quantitative strand; n = 364 principals), we apply goal theory to investigate influences on principals’ ratings in a Midwestern state in the USA. Results suggest that multiple goals may drive principals during observations. These include the following: (1) providing accurate ratings and feedback to teachers, (2) keeping teachers open to growth-promoting feedback, (3) supporting teachers’ morale and fostering positive relationships, (4) avoiding difficult conversations, (5) managing limited time wisely, and (6) maintaining self-efficacy as an instructional leader. Implications are that principals hold beneficial goals that may compete with accuracy when evaluating teachers, and that contextual differences in evaluation systems may influence the way principals act upon these goals. When responding to systemic leniency in teacher evaluations, solutions should increase accuracy in ways that minimally interfere with principals’ other beneficial goals

Principals’ severity affects teacher evaluation: statistical adjustments mitigate effects

Classroom observation is a common approach to teacher evaluation. Yet, concerns about differences in rater judgment are widespread. Despite this concern, few researchers have examined the practical impact of such differences in rater judgments on teachers’ judged effectiveness. This study fills that gap. Using data from a large-scale teacher evaluation system, we found substantial differences in principal severity that affected teachers’ classification within performance categories. We then demonstrate a technique that researchers and practitioners can use to control for differences in rater severity–thus limiting the degree to which rater severity differences can threaten the fairness of classroom observations. We discuss implications for research and practice

Students in 4\u3csup\u3eth\u3c/sup\u3e to 12\u3csup\u3eth\u3c/sup\u3e grade can distinguish dimensions of teaching when evaluating their teachers: a multilevel analysis of the TESS survey

Teacher evaluation systems across the United States are increasingly including student surveys as one source of data, but concerns remain about students’ ability to distinguish the multiple dimensions of teaching. We empirically examined this issue in an authentic evaluation system that uses a student survey based on the InTASC framework of teacher effectiveness. Our study addressed the factor structure and reliability of the Teacher Effectiveness Student Survey (TESS) using data from 5,178 surveys. We found that students’ perception of teacher effectiveness is consistent with the theory-based structure of the survey. Of particular importance, it suggests that elementary and secondary students are able to distinguish among different dimensions of teaching. Our results also indicated that TESS has a strong alignment with six InTASC-based teaching standards at both the student and teacher levels. Finally, we found that the aggregate student ratings reflect a reliable measure of the six theory-based teacher effectiveness constructs

Exploring patterns of principal judgments in teacher evaluation related to reported gender and years of experience

Meaningful interpretation of teacher evaluation based on classroom observation depends on the degree to which principals’ judgments are free from errors and systematic bias. Previous researchers have identified factors that may influence classroom observation ratings, including characteristics of students in the classrooms being observed. In this study, we explored the influence of teachers’ and principals’ characteristics on classroom observation ratings. Using data from a statewide teacher evaluation system, we examined whether principals’ ratings systematically varied across teachers’ characteristics, as well as across shared characteristics between principals and teachers. Our results suggested that principals’ ratings varied by teachers’ gender, years of experience, and school level, but not by shared demographic characteristics between principals and teachers. Principals rated female, experienced, and elementary teachers more highly than male, novice, and secondary teachers. We discuss implications for research and practice

Exploring patterns of principal judgments in teacher evaluation related to reported gender and years of experience

© 2019 Elsevier Ltd Meaningful interpretation of teacher evaluation based on classroom observation depends on the degree to which principals’ judgments are free from errors and systematic bias. Previous researchers have identified factors that may influence classroom observation ratings, including characteristics of students in the classrooms being observed. In this study, we explored the influence of teachers’ and principals’ characteristics on classroom observation ratings. Using data from a statewide teacher evaluation system, we examined whether principals’ ratings systematically varied across teachers’ characteristics, as well as across shared characteristics between principals and teachers. Our results suggested that principals’ ratings varied by teachers’ gender, years of experience, and school level, but not by shared demographic characteristics between principals and teachers. Principals rated female, experienced, and elementary teachers more highly than male, novice, and secondary teachers. We discuss implications for research and practice