Teachers’ assignments and student work: Opening a window on classroom practice

This report describes 4 years of research by the National Center for Research on Evaluation, Standards, and Student Testing (CRESST) on developing indicators of classroom practice that have the potential to be used in large-scale settings and that draw attention to important aspects of standards-based learning and instruction. CRESSTs method was based on the collection of teachers' assignments with student work. The assignments were then rated and results were summarized to create indicators of classroom practice. Results to date indicate an acceptable level of interrater reliability across study years. It likely would be necessary to collect as many as three or four assignments from teacher to obtain a stable estimate of quality. Additionally, this method was reliable when teachers created their own assignments, but not when teachers submitted assignments created by outside sources. The quality of classroom assignments was associated with the quality of observed instruction, as well as the quality of students' written work. Students who were exposed to teachers who created more cognitively challenging assignments and who had cleared grading criteria also made greater gains on the Stanford Test of Achievement, Ninth Edition (Stanford 9). The quality of teachers' assignments submitted at each of the study years, however, tended to be of basic quality only. Teachers' reactions to the data collection and implications for the use of this method in collaborative professional development sessions also are discussed. (Contains 5 figures, 1 table, and 73 references.) (Author/SLD

Measuring reading comprehension and mathematics instruction in urban middle schools: A pilot study of the Instructional Quality Assessment (CSE Technical Report 681)

The quality of reading comprehension and mathematics instruction was explored in five urban middle schools using the Instructional Quality Assessment (IQA) toolkit (N = 34 teachers). The IQA is comprised of protocols for rating observed instruction and the quality of teachers' assignments with student work. The purpose of this research was to investigate the reliability and potential validity of the ratings of these data sources. Commensurate with other research on the quality of middle schools, our results indicated that the quality of instruction varied a great deal within schools and was of a "basic" quality overall. Results indicated a moderate to high level of reliability. Four assignments with student work yielded a stable estimate of quality in both content areas, and when teachers complied with the requirements of the research as few as two observations yielded a stable estimate of teaching quality in both content areas as well. The quality of teachers' observations and assignments were significantly associated in mathematics, but not in reading comprehension. Because of the small sample size it was not possible to apply multi-level models. The relation between the IQA and student achievement on the SAT-10 was explored using linear regression techniques. Results indicated that after controlling for students' prior achievement, socio-economic status (SES), ethnicity, language, and IEP status, the IQA assignment measure in reading comprehension predicted student achievement on the Total Reading, Reading Comprehension, and Vocabulary subscores of the SAT-10. The observation measure in reading comprehension predicted student outcomes on the Reading Comprehension subscore of the SAT-10 only. In mathematics, the quality of teachers' assignments predicted students' achievement on the Procedures subscore of the SAT-10. The quality of observed instruction in mathematics predicted students' achievement on the Procedures and Total Math subscores. Without accounting for clustering within classrooms and schools as multilevel models do, our linear regression analyses may lead to results that appear stronger than they actually are. Nevertheless our analyses indicate the direction of trend in these relationships and raise important questions regarding which data sources may be best (classroom assignments or observations) for measuring specific aspects of instruction and student outcomes. Additional research with larger samples of teachers is needed to make definitive conclusions about the validity of the IQA ratings and under what conditions one might choose to either observe in classrooms or collect assignments with student work. The following are appended: (1) IQA Rubrics; and (2) High and Low Quality Examples of Observed Lessons and Assignments. (Contains 21 tables and 9 footnotes.

Utilizing Natural Language Processing for Automated Assessment of Classroom Discussion

Rigorous and interactive class discussions that support students to engage in high-level thinking and reasoning are essential to learning and are a central component of most teaching interventions. However, formally assessing discussion quality 'at scale' is expensive and infeasible for most researchers. In this work, we experimented with various modern natural language processing (NLP) techniques to automatically generate rubric scores for individual dimensions of classroom text discussion quality. Specifically, we worked on a dataset of 90 classroom discussion transcripts consisting of over 18000 turns annotated with fine-grained Analyzing Teaching Moves (ATM) codes and focused on four Instructional Quality Assessment (IQA) rubrics. Despite the limited amount of data, our work shows encouraging results in some of the rubrics while suggesting that there is room for improvement in the others. We also found that certain NLP approaches work better for certain rubrics.Comment: to be published in AIED 202

Using the Instructional Quality Assessment toolkit to investigate the quality of reading comprehension assignments and student work (CSE Report 669)

This study presents preliminary findings from research developing an instructional quality assessment (IQA) toolkit that could be used to monitor the influence of reform initiatives on students' learning environments and to guide professional development efforts within a school or district. This report focuses specifically on the portion of the IQA used to evaluate the quality of teachers' reading comprehension assignments and student work. Results are limited due to a very small sample of participating teachers (N = 13, 52 assignments), and indicate a poor to moderate level of inter-rater agreement and a good degree of consistency for the dimensions measuring academic rigor, but not the clarity of teachers' expectations. The rigor of the assignments collected from teachers also was associated with the rigor of observed instruction. Collecting four assignments (two challenging and two recent) from teachers did not yield a stable estimate of quality. Additional analyses looking separately at the two different assignment types indicate, however, that focusing on one assignment type would yield a stable estimate of quality. This suggests that the way in which assignments are collected from teachers should be revised. Implications for professional development are also discussed. The 2003 Draft Observation and Assignment Rubrics for Reading Comprehension is appended. (Contains 6 tables, 4 figures, and 4 footnotes.

Beyond summative evaluation: The Instructional Quality Assessment as a professional development tool

In order to improve students' opportunities to learn, educators need tools that can assist them to reflect on and analyze their own and others' teaching practice. Many available observation tools and protocols for studying student work are inadequate because they do not directly engage educators in core issues about rigorous content and pedagogy. In this conceptual paper, we argue that the Instructional Quality Assessment (IQA)--a formal toolkit for rating instructional quality that is based primarily on classroom observations and student assignments--has strong potential to support professional development within schools at multiple levels. We argue that the IQA could be useful to "teachers" for analyzing their own and their colleagues' practice; additionally, the IQA could aid the efforts of "principals" in their work as instructional leaders, identifying effective practitioners to help lead professional development within a school and targeting professional development needs that would require external support. Although the IQA was designed for summative, external evaluation, we argue that the steps taken to improve the reliability of the instrument--particularly the efforts to make the rubric descriptors for gradations of instructional quality as transparent as possible--also serve to make the tool a resource for professional growth among educators. The following are appended: (1) Abridged Version of the Principles of Learning; (2) Relationship between Checklist Ratings and Rubric Scores; and (3) Accountable Talk Function Checklist. (Contains 3 notes, 1 table, and 1 figure.

Using teachers’ assignments as an indicator of classroom practice (CSE Technical Report 532)

This report describes research developing indicators of classroom practice for monitoring the influence of school reform initiatives on students' learning environments, and supporting the improvement of instructional practice. The work reported here entailed collecting assignments and student work and observing classrooms in schools participating in a large-scale urban effort. Overall the reliability and consistency of the assignment ratings were good. Results also indicate that the quality of the assignments was statistically significantly associated with the quality of observed instruction and student work. It appears that our method shows promise for use in large-scale evaluation settings and identifies important dimensions of practice that could support teacher self-evaluation and reflection

Creating high-quality classroom assignments

"Research indicates that the quality of classroom assignments makes a difference in student learning. Moreover, assignment quality varies between teachers, even within the same school. To support teachers in developing, reflecting on, and fine-tuning assignments, Creating High Quality Classroom Assignments presents a series of rubrics with benchmark examples from elementary, middle, and high school classrooms. These rubrics serve as diagnostic tools to assess the strengths and weaknesses of assignments and guide the creation of new ones. Questions are also provided at the end of each chapter to direct reflection on a specific dimension of assignment quality." "This book will be useful to teachers for self-assessment purposes, collaborative professional development settings, or as a basic text in teachers education courses." -Book Jacke