32 research outputs found
7. Legal and Professional Issues in Teacher-Certification Testing: A Psychometric Snark Hunt
The validation of teacher-certification tests has become a snark hunt. The snark pursued by test contractors however, is not nearly as elusive, nor fanciful as Carroll\u27s shadowy beast. Their snark is a legally defensible test. Test contractors are-merely from nervousness (not from goodwill)-marching along shoulder to shoulder with jurists. I will argue that hunting the shadowy snark through the dark wood of the court room cheapens the conceptualization of validity and delimits the conduct of validation studies to minimalist exercises designed to obtain a positive result.
I start from the basic premise that the most important feature of any test is the accuracy of the inferences or decisions made from the score. Countless generations of measurement students have been taught the old saw: A test can be reliable but not valid, but if it\u27s valid, it has to be reliable. Here I argue that a test can be legally defensible but not valid. However, if the results from a well designed validation process, on balance support the inferences or decisions made on the basis of the test, legal defensibility should be satisfied. Legal defensibility is important. Tests must be able to stand court scrutiny. However, I contend, we have the cart before the horse-whatever courts will accept drives current validation practices. Legal precedent is circumscribing inquiry about the essential question underlying all testing- how correct are the inferences made from a person\u27s score?
The vulnerability of testing companies engaged in this legal snark hunt consists in their readiness to view their task no more broadly than that of satisfying the desire of state agencies for legally defensible certification tests. Some contractors are in danger of becoming ambidextrous yes men for states. Testing companies, of course, insist on test validity. However, faced with the exigencies of the applied world- legal precedent, contract obligations, tenacious timelines, and budget limitations-they offer excuses to absolve themselves from gathering construct and criterion-related evidence related to their products. They have built a tunnel through a mountain of precepts about validity which are regularly taught students in test and measurement classes. They take the law of validity literally, but argue that the letter is elastic, pleading for leniency from basic precepts and strictures. The companies deny that their product purports to measure a complex construct- a subset of competence in the classroom.
This behavior of test contractors is a reaction to the following factors:
• the mandates of policy makers for a quick, visible, quantifiable, and administratively convenient fix to a perceived problem with the quality of the teaching corps • a naive confidence on the part of policy makers and the public that multiple-choice tests can improve the quality of the teacher corps • a lack of understanding on the part of policy makers and the public, concerning the kinds of evidence needed to support the inferences they wish to make about teacher candidates • the bureaucratic nature of the state agencies charged with implementing the mandate of policy makers • the funding level and lead time available to implement a systematic validation effort • the applied, commercial nature of the testing industry • the selection of contractors through the competitive, low-bid RFP process • the allegiance of testing companies to the state agency that awards the contract, rather than to examinees who later pay to sit the exam, or the general public who often base their perceptions of educational quality on test scores • the evolution of a set of tried and true procedures for gathering content validation evidence that redounds to confirmation rather than disconfirmation • the vested interest of policy makers in vigorously defending their programs when legally challenged • the competitive, adversarial nature of litigation that colors arguments on both sides about scientific-technical issues • the existence of legal precedent about what is sufficient to justify the use of employment tests
I was asked to address the professional and legal implications of teacher-certification testing. This chapter, therefore, consists of two distinct sections. In the first section, I examine the legal challenges to teacher-certification tests. I offer examples of how testing experts operate within the legal arena. In this first section validity issues are interpreted from the legal rather than psychometric tradition. I begin by describing two types of legal challenges to teacher- certification tests and illustrate the part experts from the psychometric community play in these legal challenges. I conclude the first section with a description of the formula for a legally defensible teacher-certification test that has emerged from court decisions.
In the second section I leave behind the legal tradition and examine the validity issues surrounding teacher certification testing from the professional point of view. I begin by offering examples of the inferences and decisions made by various publics from scores on teacher certification. I then develop the psychometric implications for validation that flow from these inferences, and the problems associated with them in the applied situation. Next, I discuss issues related to the mix of content, construct, and criterion-related evidence that I feel is necessary if we are to understand what we are measuring and \u27 basing important certification decisions on. Finally, because the cut-score is the trigger for any inference or decision about teachers, I raise issues related to its validity.
Before I begin, two caveats. First, my treatment of litigation is strictly that of layman. My credentials consist of a single week in law school before leaving for the army, a decision I\u27ve never regretted. Second, I have a bias about high-stakes tests. If the use of a test has the potential to harm, or has serious consequences for individuals, the test should be subjected to a thoroughgoing validation before it is used operationally. Teacher-certification tests certainly trigger this bias. Large numbers of examinees are affected, particularly large numbers of minorities. From my perspective, the protection of the individual takes precedent over the state\u27s interests until the state can demonstrate that the test, although not error free, does a reasonably good job of identifying individuals who in all likelihood lack the necessary skills or knowledge to be minimally successful in the classroom. The last clause anticipates my later argument that teacher certification tests are really about a candidate\u27s competence in the classroom
Madaus, George F., The Influence of Testing on Curriculum, pp. 83-121 in Laurel N. Tanner, ed., Critical Issues in Curriculum. 87th Yearbook of the NSSE, Part I. Chicago: The University of Chicago Press, 1988. Reprinted pp. 73-111 in Margaret J. Early and Kenneth J. Rehage, eds., Issues in Curriculum: A Selection of Chapters from Past NSSE Yearbooks. 98th Yearbook of the NSSE, Part II. Chicago: The University of Chicago Press, 1999.
Reviews the effects of testing, especially high-stakes standardized tests, on curriculum, instruction, and students; discusses past and present issues with examples from several countries
Evaluation models : Viewpoints on educational and human services evaluation
xii, 423 p.; 21 cm
Harvard educational review
Publ. dans la revue Harvard educational review, vol. 49, no 2 (mai 1979), p. 207-230
Madaus, George F., and Thomas Kellaghan, Curriculum Evaluation and Assessment, pp. 119-154 in Philip W. Jackson, ed., Handbook of Research on Curriculum. New York: Macmillan, l992.
Summarizes the state of evaluation and assessment in curriculum, including historical issues, issues in testing, and six major components of curriculum evaluation
Madaus, George, Michael Russell, and Jennifer Higgins, Why and How Should High-Stakes Testing be Monitored? pp. 197-221 in their The Paradoxes of High-Stakes Testing. Charlotte, NC: Information Age Publishing, 2009.
Traces efforts made to monitor the technical aspects of tests for users and policymakers; proposes an independent body to perform this oversight function and suggests eight principles to follow in creating such an ongoing institution
Madaus, George F., and Laura M. O\u27Dwyer, A Short History of Performance Assessment: Lessons Learned, Phi Delta Kappan, 80(May, 1999), 688-695.
Traces the history of performance assessment from 210 BC to the present
Consistency of Findings Across International Surveys of Math & Science
The investigation reported in here was prompted by discrepancies between the performance of Irish students on two international tests of science achievement: the Second International Assessment of Educational Progress (IAEP2) administered in 1991 and the Third International Mathematics and Science Study (TIMSS) administered in 1995. While average science achievement for Irish 13-year-olds was reported to be at the low end of the distribution representing the 20 participating countries in IAEP2, it was around the middle of the distribution representing the 40 or so countries that participated in TIMSS at grades 7 and 8. An examination of the effect sizes associated with mean differences in performance on IAEP2 and TIMSS indicated that the largest differences are associated with the performance of students in France, Ireland and Switzerland. Five hypotheses are proposed to account for the differences