Implementation Fidelity and Attainment in Computerized Practice of Mathematics

Measuring the implementation fidelity (IF) or integrity of interventions is crucial, otherwise a positive or negative outcome cannot be interpreted. Direct and indirect methods of IF measurement tend to over-emphasize teacher behaviour. This paper focuses on IF measured by student behaviour collected through computers. Attainment was measured by the STAR test of maths (a computerized item-banked adaptive norm-referenced test). Implementation quality (IF) was measured by Accelerated Maths (AM) (an instruction-free personalized practice and progress-monitoring system in mastery of mathematics skills). Attainment data was gathered in the UK on 20,103 students in 148 schools, and of these implementation data on n=6,285. Only a small percentage of pupils scored on five AM implementation indices at or above the levels recommended. Correlations between attainment and implementation indices were modest, but high implementation was positively correlated with high attainment. Socio-economic status did not appear to affect implementation or attainment. Implementation quality of AM is clearly a problem in the UK, and needs improvement. However, overall students still scored above average on attainment

Efficiency of Targeted Multistage Calibration Designs Under Practical Constraints: A Simulation Study

Calibration of an item bank for computer adaptive testing requires substantial resources. In this study, we investigated whether the efficiency of calibration under the Rasch model could be enhanced by improving the match between item difficulty and student ability. We introduced targeted multistage calibration designs, a design type that considers ability‐related background variables and performance for assigning students to suitable items. Furthermore, we investigated whether uncertainty about item difficulty could impair the assembling of efficient designs. The results indicated that targeted multistage calibration designs were more efficient than ordinary targeted designs under optimal conditions. Limited knowledge about item difficulty reduced the efficiency of one of the two investigated targeted multistage calibration designs, whereas targeted designs were more robust