Reliability and Criterion Validity of Two Algebra Measures: Translations and Content Analysis-Multiple Choice

This technical report summarizes the results of a study in which we examined the technical adequacy of two potential measures for algebra progress monitoring. Eighty-seven students (11 of whom were receiving special education services) completed two forms of a Translations measure and two forms of a Content Analysis-Multiple Choice measure during each of two data collection sessions. In addition, we gathered data on criterion variables including grades, overall grade point average, teacher ratings of student proficiency, and scores on districtadministered standardized tests, as well as a measure of algebra aptitude. We examined both test-retest and alternate form reliability for both single probe scores and aggregated scores (computed by averaging two individual scores). Criterion validity was examined by computing correlations between students’ single and aggregated scores on the probes with their scores on other indicators of proficiency in algebra. The results of this study suggest that the Translations measure is more promising than the Content Analysis-Multiple Choice measure in terms of both reliability and criterion validity. The strength of the relations obtained in this study were in the low to moderate range and were not as strong as the relations obtained with a different sample in this district using three other algebra measures (see Project AAIMS Technical Report 2 for details of the earlier study). Both measures produced acceptable distributions that were free from floor and ceiling effects. Students had roughly similar means and standard deviations on both measures. Reliability estimates for both measures fell short of expected levels for both single probes and aggregated scores. The Translations measure produced stronger correlations than the Content Analysis-Multiple Choice measure, but did not demonstrate a level of reliability that would be acceptable for instructional decision making. The majority of the criterion validity relations were in the low to moderate range. Aggregated scores produced improvements in the criterion validity estimates for the Translations measure, but not for the Content Analysis-Multiple Choice measure. The strongest relations were identified between the Translations measure and eighth graders’ performance on the district’s math achievement test, as well as between the Translations measure and all students’ performance on the algebra aptitude test. These two relations were in the moderate to strong range; relations between the Translations measure and the remaining criterion variables were in the low range

Alignment of Algebra Curriculum, Assessment, and Instructional Practices in District B: A Case Study of Fall 2004

Project AAIMS (Algebra Assessment and Instruction: Meeting Standards) is a federally funded project that has two objectives. The first is to examine the alignment of algebra 1 curriculum, instruction, and assessment in general and special education. The second is to develop and validate algebra assessment tools for use in general and special education classes. This case study focuses on the first objective – it examines the alignment of algebra curriculum, instruction, and assessment for students with and without disabilities in one of the three districts participating in Project AAIMS

Classroom Observation Data for District B: Anecdotal Observation Results

This report documents the results of anecdotal observations conducted in District B during the fall of 2004. It describes the algebra topics addressed during our observations, the expected tasks (class activities), teacher actions, and student actions in four Algebra IA classes and two Algebra IB classes in this district. We looked at the algebra curriculum for students in these beginning algebra classes, the ways that class periods were structured in these classes, the kinds of instructional approaches that were used, and students’ responses to these instructional approaches. Student with and without disabilities were all enrolled in general education beginning algebra classes in District B; therefore they completed the same curriculum. The two Algebra IA teachers moved through the textbook at slightly different rates, but students were exposed to basically the same content. One teacher taught both of the Algebra IB classes, and her lessons concentrated on the same topics for each class. The most common expected task varied by teacher. Teacher 1 taught one section of Algebra IA and two sections of Algebra IB. In her Algebra IA class, the most typical task was checking homework, in Algebra IB, it was leading a review. Teacher 2 taught three sections of Algebra IA where the most prevalent expected task was teacher-led instruction. The most typical instructional approaches that we observed in District B were providing individual student assistance and modeling as the teacher showed how to solve algebra problems or reviewed for an exam in both courses. Completing assignments was the most typical productive student action in both courses, with listening observed just as often in the Algebra IB classes. Off task behavior was the most common nonproductive student action, and it was the most often observed student action in District B

Classroom Observation Data for District C: Momentary Time Sampling

This report documents the results of momentary time sampling observations conducted in District C during the fall of 2004. It identifies typical student and teacher behaviors, as well as typical instructional organization patterns and task formats in Algebra I classes in this district. We found that District C beginning algebra teachers devoted nearly equal amounts of class time to whole class and independent work. These teachers spent about one half of the time we observed engaged in talking to their students about algebra or listening to students’ questions or comments about the day’s lesson. Their students were assigned paper and pencil tasks for more than half of the observational intervals and were expected to listen to lectures or participate in discussions for slightly more than 40% of the time. The most typical student behavior was listening to teachers (or displaying some other appropriate behavior) with taking notes, working on an assignment, or answering a teacher’s question (all active academic responses) as the second most typical type of student behavior

Reliability and Criterion Validity of Five Algebra Measures in Districts B and C

This technical report summarizes the results of a study in which we examined the technical adequacy of five potential measures for algebra progress monitoring. One hundred three students (14 of whom were receiving special education services) completed two forms of a Basic Skills measure, two forms of an Algebra Foundations measure, one form of a Content Analysis-Constructed Response measure, two forms of a Translations measure, and two forms of a Content Analysis-Multiple Choice measure administered over two data collection sessions. Each probe data collection session was repeated to investigate the test-retest reliability of the measures. In addition, we gathered data on criterion variables including grades, overall grade point average, teacher ratings of student proficiency, and scores on district-administered standardized tests, as well as a measure of algebra aptitude. We examined both test-retest and alternate form reliability for both single probe scores and aggregated scores (computed by averaging two individual scores). Criterion validity was examined by computing correlations between students’ single and aggregated scores on the probes with their scores on other indicators of proficiency in algebra. We found that four of the five measures produced effective distributions of student scores, with no signs of floor or ceiling effects. On the Translations probe, students produced nearly as many incorrect responses as they did correct responses, suggesting a high rate of guessing on that measure. The test-retest and alternate form reliability of single probes ranged from .4 to .9, with most coefficients in the .4 to .6 range. Aggregating scores from two probes produced slight increases in the reliability of the probes, with most correlations ranging from .5 to .7. For both single probes and aggregated scores, test-retest reliability coefficients exceeded those obtained for alternate form reliability. Neither the single nor the aggregated probes consistently produced reliability coefficients above the .80 level that represents a standard benchmark. Criterion validity coefficients were also lower than those obtained in previous research (Foegen & Lind, 2004). Coefficients were generally in the low range (.2 to .4); the exception to this pattern was for the Iowa Algebra Aptitude Test, which was more strongly related to the algebra progress monitoring measures (coefficients in the .3 to .5 range). The Content Analysis Constructed Response, the Algebra Foundations, and the Content Analysis-Multiple Choice measures produced the strongest relations with the criterion measures, with lower relations obtained for the Basic Skills and Translations measures. Concerns were identified with difficulty of scoring the Content Analysis-Constructed Response probes efficiently and accurately, which will likely limit the viability of this measure in applied settings. Issues for future research are identified

Classroom Observation Data for District A: Anecdotal Observation Results

This report documents the results of anecdotal observations conducted in District A during the spring of 2004. It describes the algebra topics addressed during our observations, the expected tasks (class activities), teacher actions, and student actions in six different beginning algebra courses this district. We looked at the similarities and differences in the algebra curriculum for students with and without disabilities in the different algebra courses, the ways that class periods were structured in these classes, the kinds of instructional approaches that were used in general education and special education algebra courses, and students’ responses to these instructional approaches