The influence of changes in assessment design on the psychometric quality of scores (Research Memorandum RM-00-1). Princeton, NJ: educational Testing Service

Abstract This article addresses the problem of improving the measurement quality of a complex performance assessment through principled assessment design. We describe the characteristics and measurement impact of steps taken to improve assessment exercise design along with modifications in assessor training materials and procedures between the 1995-1996 and the 1996-1997 administrations of the National Board for Professional Teaching Standards Early Childhood/Generalist examination. Specifically, we describe how the revision of this assessment resulted in increases in the inter-assessor agreement, internal consistency, and generalizability of scores. All indices we examined improved as a result of the revisions. The results suggest that previously observed limits on the measurement quality of performance assessments due to the relatively small number of items that contribute to an assessment score can be altered significantly through attention to assessment design and related scoring processes. Far too often at this relatively early stage tasks are "created" and then rationalized rather than carefully and systematically designed. More interestingly, design processes can influence external validity criteria, that is, how performance-based assessments perform. (p.99) The absence of principled design has led to assessments that have been challenged as not having the psychometric qualities to justify high-stakes decisions (e.g., However, low covariance among performance assessment tasks may also be a problem. For example, In this article, we suggest that reliability can be influenced substantially by improving the design and scoring of performance assessments. This study explores whether indices of psychometric quality can be improved when design and scoring decisions are made with conscious attention to evidential issues raised by Messick (1989), An adequate performance assessment design must address the following two questions satisfactorily: 1. Does the information provided in a response (the answer) lead to interpretations by assessors that are both consistent and relevant to the intended purpose of the item? 2. Does the assessment, taken as a whole, provide coherent evidence that supports one or more target inferences consistent with the purpose of the assessment? Focusing on these questions, we discuss progress that has been made in the design and development of a high-stakes assessment for the National Board for Professional Teaching Standards (NBPTS, 1997(NBPTS, , 1998. These assessments consist of a relatively small number (10) of complex assessment tasks that yield a great deal of performance evidence within a task, but that are summarized by a single score per task. Thus, since the amount of information (number of scores) is low by assessment standards, it is imperative that the pieces of information provide the clearest evidence and most coherent inferences about an individual as possible. To satisfy this goal, we have placed great effort in the overall design of the assessments, particularly in the areas of item development and scoring processes. This article reviews how changes in these aspects of the assessment impact the psychometric quality of the resulting scores. The National Board Assessments The mission of the National Board for Professional Teaching Standards is to institute a national certification system that allows teachers to demonstrate accomplishment of high standards in the teaching profession and through which the teaching profession is enhanced. To this end, the National 3 Board has developed several certification examinations based on specific professional criteria. Teachers who volunteer to attempt to attain National Board certification are required to complete a series of performance-based assessments that ask teachers to include videotapes of classroom instruction, examples of instructional materials and student work, all with extensive written commentary, in addition to completing a set of essays in a timed assessment center setting. All assessments are based on NBPTS Standards written for teachers of specific content and students from specific age ranges