Accessed 23,426 times on https://pareonline.net from December 07, 2001 to December 31, 2019. For downloads from January 1, 2020 forward, please click on the PlumX Metrics link to the right

Kellow, J. Thomas

Willson, Victor L.

English

ScholarWorks@UMass Amherst

Practical Assessment, Research, and Evaluation Volume 7 Volume 7, 2000-2001 Article 24 2000 Consequences of (mis)use of the Texas Assessment of Academic Skills (TAAS) for high-stakes decisions: A comment on Haney and the Texas miracle in education J. Thomas Kellow Victor L. Willson Follow this and additional works at: https://scholarworks.umass.edu/pare Recommended Citation Kellow, J. Thomas and Willson, Victor L. (2000) "Consequences of (mis)use of the Texas Assessment of Academic Skills (TAAS) for high-stakes decisions: A comment on Haney and the Texas miracle in education," Practical Assessment, Research, and Evaluation: Vol. 7 , Article 24. DOI: https://doi.org/10.7275/yn8g-gj30 Available at: https://scholarworks.umass.edu/pare/vol7/iss1/24 This Article is brought to you for free and open access by ScholarWorks@UMass Amherst. It has been accepted for inclusion in Practical Assessment, Research, and Evaluation by an authorized editor of ScholarWorks@UMass Amherst. For more information, please contact scholarworks@library.umass.edu. A peer-reviewed electronic journal.Copyright is retained by the first or sole author, who grants right of first publication to Practical Assessment, Research & Evaluation. Permission is granted to distribute this article fornonprofit, educational purposes if it is copied in its entirety and the journal is credited. PARE has the right to authorize third party reproduction of this article in print, electronic anddatabase forms.Volume 7, Number 24, December, 2001 ISSN=1531-7714Consequences of (mis)use of the Texas Assessment of Academic Skills(TAAS) for high-stakes decisions:  A comment on Haney and the Texasmiracle in education.J. Thomas Kellow, University of HoustonVictor L. Willson, Texas A&M UniversityAbstractThis brief paper explores the consequences of failing to incorporate measurement error inthe development of cut-scores in criterion-referenced measures.  The authors use the caseof Texas and the Texas Assessment of Academic Skills (TAAS) to illustrate the impact ofmeasurement error on “false negative” decisions.  These results serve as further evidenceto support Haney’s (2000) contentions regarding the (mis)use of high-stakes testing in thestate of Texas.Walt Haney’s (2000) treatise on The Myth of the Texas Miracle in Education highlights a number of concerns related tohigh-stakes decision-making in K-12 settings.  His elaboration of the history and development of the Texas Assessmentof Academic Skills (TAAS) was illuminating, especially for those unfamiliar with the often capricious fashion in whichstandards are ultimately set for high-stakes decisions.  The evidence Haney presents in evaluating the TAAS fits wellwith Samuel Messick’s (1994) argument for considering the consequences of test use.  Although sometimes referred to asconsequential validity, Messick considered this aspect of test interpretation and use to be one of many forms of constructvalidity.  His view of the evaluative role of validity is summarized nicely in the following paragraph:When assessing any of these constructs – whether attributes of persons or groups, of objects or situations –validity needs to be systematically addressed, as do other basic measurement issues such as reliability,comparability, and fairness.  This is so because validity, comparability, and fairness are not justmeasurement principles, they are social values that have meaning and force outside of measurementwhenever evaluative judgment and decisions are made.  As such, validity assumes both a scientific andpolitical role that can by no means be fulfilled by a simple correlation coefficient between test scores and apurported criterion or by expert judgments that test content is relevant to the proposed test use (p. 1).One technical aspect of the TAAS that Haney takes to task is the reliability of scores yielded by the test and thepotential for misclassification as a function of measurement error.  As the author notes, “The reason the setting ofpassing scores on a high-stakes test such as the TAAS is so important is that the passing score divides a continuum ofscores into just two categories, pass and fail.  Doing so is hazardous because all standardized test scores contain somedegree of measurement error”  (Haney 2000, p. 10).  Haney also points out that measures of test-retest (or alternate-form) reliability and not internal consistency should ideally be used to inform judgment as to the potential formisclassfication due to measurement error.  In light of the conspicuous absence of test-retest data on the TAAS, heattempts to use extant data to approximate this reliability estimate as compared to the internal consistency (KR20)estimates provided by the Texas Education Agency (TEA).  Although the approach he uses is somewhat problematic (seeWainer, 1999), it is clear that these test-retest estimates are lower than the KR20 estimates.The thrust of Haney’s argument is that measurement error inherent in the TAAS (or any other measure, for thatmatter) contributes appreciably to the rate of “false negatives,” or misclassifying passing students as non-passing.  Hisfocus, however, is exclusively on the tenth-grade Exit-Level TAAS since this is the test high school students in the stateof Texas must pass in order to graduate from high school (although it is not the sole criterion).  But in some districts, theTAAS is also being used at other grade levels for high-stakes decisions – namely, promotion and retention of students. Waco Independent School District and Houston Independent School District, among others, require students in grades 3through 8 to pass all portions of the TAAS in order to be considered for promotion to grade level.  Students who fail anysubtest during the statewide spring administration can expect to attend mandatory summer school, after which they aretested on a “released” version of the TAAS.  Those students who fail are held back in grade.  This stands in contrast toPage 1 of 5 1Kellow and Willson: Consequences of (mis)use of the Texas Assessment of Academic SkilPublished by ScholarWorks@UMass Amherst, 2000the Exit-Level testing schedule where, as Haney notes, students have as many as eight opportunities to pass the exam. In addition, the TEA has announced plans to use the TAAS for promotion/retention decisions in the third, fifth, andeighth grades beginning in the fall of 2001 (Texas Education Agency, 1999).Given the current political climate and the clarion calls for school district accountability throughout the nation, thenumber of districts in Texas using the TAAS for promotion and retention decisions in all tested grades no doubt willincrease.  Indeed, the Houston model has been lauded for its strict criteria for student grade promotion, which is believedto increase student motivation and achievement while reducing the student dropout rate (Markley, 1999).  Our purposein this response is to elaborate on the potential social consequences of using the TAAS for high-stakes decisions,specifically grade promotion and retention.  The question we asked is rather simple:  “Given the imperfect reliability ofthe TAAS test, how many students with a true passing score are potentially misclassified as failing across grades andsubtests in a given year”?   Fortunately, sufficient summary statistics and frequency distributions were available fromthe TEA to estimate these numbers. MethodThe most recent reliability estimates for the TAAS were reported for the 1998-99 school year; therefore we used means,standard deviations, and frequency distributions for this same year.  We want to emphasize that there are a number ofways to go about this estimation process. The method presented emerged because it is fairly intuitive and requiredminimal summary statistics from the data. First, we should note that the 70% passing standard on the TAAS mentioned by Haney is not fixed across grades andsubtests.  Because of differential item and thus test difficulty, the proportion of items correct needed to pass a givensubtest ranges from .64 to .75, according to the TEA.  The Texas Learning Index (TLI) was developed by the TEA for thepurpose of, among other things, providing a consistent passing standard across test forms.  The TLI is a linearstandardized scoring transformation with a standard deviation of 15  and an anchor (rather than mean) of 70, whichrepresents the passing standard for a given subtest at a given grade.  The TLI is calculated in z-score form as:TLI= [(z observed - z passing)*15] + 70 (1)Although the TLI observed score of 70 is the passing standard, this standard fails to incorporate measurement error indetermining the appropriate cut score.  Specifically, the process of modifying cut-scores involves determining the domainscore in proportion-correct form that constitutes mastery (to ) and then adjusting this value to estimate a new cut score(X0) in number-correct form that reflects the measurement error in the test data (Crocker & Algina, 1986).  Huynh andSanders (1980) provide an approximate procedure for this purpose that works well when a test consists of 20 or moreitems and the observed proportion correct cut score falls within .50 to .80. The TAAS subtests meet both criteria.  Thisformula is given as:(2)Where n is the number of items on the test, to is the observed proportion-correct cut score, and mx is the mean number ofitems correct.  As noted by Crocker and Algina (1986), as KR21 approaches 1.0, X0  approaches n to + .5 irrespective ofthe value of mx. Our question focuses on the estimate X0  when transformed to a TLI score metric.  Put simply: What is the passing TLIadjusted for measurement error for a given subtest in a given grade, and what percent and number of students met thisadjusted criterion but not the standard cut score of 70?  The following steps were employed to determine X0 in TLI form:1. calculate X0 in raw score form;2. transform X0 into a z-score;3. substitute z X0 for z observed in Formula 1.ResultsTable 1 provides the adjusted cut score X0  in the TLI metric across grades for both the mathematics and readingsubtests.Page 2 of 5 2Practical Assessment, Research, and Evaluation, Vol. 7 [2000], Art. 24https://scholarworks.umass.edu/pare/vol7/iss1/24DOI: https://doi.org/10.7275/yn8g-gj30Table 1TLI passing cut-scores adjusted for measurement error by subtestand grade  Grade Reading Mathematics3rd 67.5 67.44th 67.2 67.65th 67.2 67.06th 67.6 68.27th 67.9 68.48th 67.7 68.310th 66.9 68.7   We then used TLI frequency distributions for the 1998-99 administration of the TAAS to determine the percent andnumber of students who received a TLI score of X0  or higher but less that 70.  Because the TLI frequency distributiontables obtained from TEA report whole number values, we rounded the obtained X0 estimates to the closest wholenumber.  These data are presented in Table 2 disaggregated by subtest and grade.Table 2Percent and number of potentially misclassified students bysubtest and grade  Grade Reading Mathematics3rd 1.7 (n=4,243) 2.9 (n=7,151)4th 1.7 (n=4,105) 2.1 (n=5,239)5th 2.2 (n=5,509) 2.4 (n=6,007)6th 2.2 (n=5,725) 2.5 (n=6,653)7th 2.0 (n=5,410) 2.8 (n=7,474)8th 1.4 (n=3,620) 2.6 (n=6,903)10th 2.9 (n=6,570) 1.6 (n=3,650)Because of the rounding procedure mentioned earlier, these percentages and student numbers are approximations. Roughly 35,182 students who took the reading subtest in the 1998-99 school year were classified as failing, despitehaving an observed score that would have met (or exceeded) the passing criterion had the presence of measurementerror been incorporated into the cut score.  On the mathematics subtest, 43,077 students who failed met (or exceeded)Page 3 of 5 3Kellow and Willson: Consequences of (mis)use of the Texas Assessment of Academic SkilPublished by ScholarWorks@UMass Amherst, 2000the adjusted observed criterion score. Discussion     Because all tests are inherently unreliable to some degree, measurement errors must be accommodated in thedevelopment of cut-scores for criterion-referenced tests, particularly when these instruments are used to make high-stakes decisions for student placement.  Our analysis focused exclusively on the impact of false negative classificationerrors.  There exists, of course, a second type of misclassification termed a “false positive,” or misclassifying non-passingstudents as passing.  Although both types of misclassification are serious, a survey of Texas educators conducted byHaney (2000) as part of his investigation of the TAAS  indicated that respondents viewed the consequences of denying ahigh school diploma to a qualified student based on a classification error (false negative) as considerably more seriousthat granting a diploma to an unqualified student (false positive).  Indeed, only the consequences to society of granting alicense to an unqualified pilot, physician, or teacher, respectively, were viewed as more serious.  Additionally, theliterature on grade retention is fairly consistent in noting the deleterious effects of these policies (e.g., increased dropoutrates), particularly when strong individualized remediation procedures are not in place (McCoy & Reynolds, 1999).  Putsimply, based on Haney’s (2000) survey and the empirical findings on the consequences of retaining students, we feel itseems reasonable to place greater emphasis on the occurrence of false negatives -- at least in the context of educationand student promotion decisions.     The estimation process we employed produced results suggesting that about 2% of students who take the state-mandated TAAS exam will be scored as false negatives on one or more of the subtests.  The consequences ofmisclassification will become more evident as the TAAS is increasingly used for promotion and retention decisions inTexas.  There is, however, a much larger picture emerging at the national level regarding the (mis)use of standardizedassessment tools.  To put this in a broader perspective, we extended our analysis to include an estimate of how manystudents nationally would be misclassified as false negatives if a testing program such as the TAAS were in place.  Weassumed that testing would include the same grade levels as the Texas accountability model, and assumed also anational testing instrument with the same technical adequacy (reliability) as the TAAS.  National data were obtainedfor the 1998-99 school year disaggregated by grade level.   Combining both reading and mathematics error rates resultsin approximately  1.1 million students that would potentially be misclassified as false negatives each year across thecountry.  Two percent clearly is no small number in the national context.             The recently installed Bush administration has issued school accountability reform measures that rely almostexclusively on standardized achievement tests.  Many questions remain, however, regarding the structure andimplementation of the testing program that will serve as a measure of student performance across states.  What thesetests will look like, the extent to which they yield scores that are psychometrically meaningful, and the importance of thescores in guiding student-level decisions are issues that have not been addressed to date.   It is notable that both theAmerican Psychological Association (APA) and, more recently, the American Educational Research Association (AERA)have issued position statements advising against the use of a single assessment for high-stakes decisions at theindividual level.  It seems probable, however, that the national appetite for school accountability in the form of studentachievement scores will overwhelm any concerns over the ethical consequences of high-stakes testing. ReferencesCrocker, L., & Algina, J.  (1986).  Introduction to classical and modern test theory.  Orlando:  Harcourt Brace.Haney, W. (2000).   The myth of the Texas Miracle in education.  Education Policy Analysis Archives [On-line serial], 8(41).  Available:  http://epaa.asu.edu/epaa/v8n41/.Huynh, H., & Saunders, J. C.  (1980).  Accuracy of two procedures for estimating reliability of mastery tests.  Journal ofEducational Measurement, 17, 351-358.Markley, M.  . (1999, September 8). HISD rules holding more students back:  Expanded standards cut social promotions. Houston Chronicle, p. A1.Messick, S. A.  (1994).  Foundations of validity:  Meaning and consequences in psychological assessment.  EuropeanJournal of Psychological Assessment, 10, 1-9.McCoy, A. R., & Reynolds, A. J.  (1999).  Grade retention and school performance:  An extended investigation.  Journalof School Psychology, 37, 273-298.Texas Education Agency (1999, June 22).  Briefing book:  Legislation affecting public education [On-line].  Available: http://www.tea.state.tx.us/brief/doc2.html.Wainer, H. (1999).   Comments on the ad hoc committee’s critique of the Massachusetts Teacher Tests.   EducationPage 4 of 5 4Practical Assessment, Research, and Evaluation, Vol. 7 [2000], Art. 24https://scholarworks.umass.edu/pare/vol7/iss1/24DOI: https://doi.org/10.7275/yn8g-gj30Policy Analysis Archives [On-line serial], 7 (5).  Available:  http://epaa.asu.edu/epaa/v7n5.html.  Descriptors: Test Validity; Test reliability; Consequences; Impact; ErrorCitation: Kellow, J. Thomas & Victor L. Willson (2001). Consequences of (mis)use of the texas assessment of academic skills (taas) for high-stakes decisions: a comment on haney and the texas miracle in education. Practical Assessment, Research & Evaluation, 7(24). Availableonline: http://PAREonline.net/getvn.asp?v=7&n=24.Page 5 of 5 5Kellow and Willson: Consequences of (mis)use of the Texas Assessment of Academic SkilPublished by ScholarWorks@UMass Amherst, 2000

Consequences of (mis)use of the Texas Assessment of Academic Skills (TAAS) for high-stakes decisions:  A comment on Haney and the Texas miracle in education

Consequences of (mis)use of the Texas Assessment of Academic Skills (TAAS) for high-stakes decisions: A comment on Haney and the Texas miracle in education

Abstract

Similar works

Full text

Available Versions

ScholarWorks@UMass Amherst