Differential Item Functioning in a Teacher-Created Benchmark Mathematics Assessment

Abstract

The purpose of this study was to examine the presence of gender and ELL Differential Item Functioning (DIF) in a teacher-created mathematics benchmark test in one public charter school district in Northeast Georgia. DIF occurs when an item behaves differently in different subgroups, rather than measuring a test taker’s true ability. The geometry assessment used in this study consisted of 34 dichotomously scored items, and the examinees (N = 183) were a mix of ninth-, tenth-, and eleventh-grade students. The study obtained and analyzed data in a convergent sequential design in two phases using the Delphi and Mantel-Haenszel methods. The first stage sought to find consensus amongst Delphi panel members on the items and their attributes (dis)favoring (a) male and female students and (b) ELL and non-ELL students. The second stage sought to find the amount of statistical DIF using the thin and thick matching variables of the Mantel-Haenszel method. A Delphi panel of 10 experts consented on 31 out of 34 items on gender DIF, and consented on 34 out of 34 items on ELL DIF with a high degree of consensus and a high degree of internal consistency. The panel found that 30% of items had high degree of ELL DIF items; however, no item displayed a high degree of gender DIF. The Mantel-Haenszel analysis of 183 student scores revealed that 10 out of 34 items exhibited ELL DIF and 4 items exhibited gender DIF. Finally, the researcher combined the results from the Delphi and Mantel-Haenszel analyses to explain the DIF phenomena in the teacher-created benchmark mathematics assessment. Critical feminist theory and critical race theory inform the philosophical approach to the problem of DIF. Although many national- and state-level exams are evaluated for DIF, most classroom exams are not. Given the importance of these tests in student assessment, placement, and access to opportunity, it is imperative that teacher-created tests do not discriminate against students belonging to particular gender or ethnic groups. Thus, this study makes recommendations on how to avoid building DIF into assessments