50 Years of Test (Un)fairness: Lessons for Machine Learning

Quantitative definitions of what is unfair and what is fair have been introduced in multiple disciplines for well over 50 years, including in education, hiring, and machine learning. We trace how the notion of fairness has been defined within the testing communities of education and hiring over the past half century, exploring the cultural and social context in which different fairness definitions have emerged. In some cases, earlier definitions of fairness are similar or identical to definitions of fairness in current machine learning research, and foreshadow current formal work. In other cases, insights into what fairness means and how to measure it have largely gone overlooked. We compare past and current notions of fairness along several dimensions, including the fairness criteria, the focus of the criteria (e.g., a test, a model, or its use), the relationship of fairness to individuals, groups, and subgroups, and the mathematical method for measuring fairness (e.g., classification, regression). This work points the way towards future research and measurement of (un)fairness that builds from our modern understanding of fairness while incorporating insights from the past.Comment: FAT* '19: Conference on Fairness, Accountability, and Transparency (FAT* '19), January 29--31, 2019, Atlanta, GA, US

Educational policy analysis archives

https://digitalcommons.usf.edu/usf_EPAA/1378/thumbnail.jp

Advanced Placement: Access Not Exclusion

Lichten (2000) argues that increased access to AP courses in high schools has led to a decline in AP quality. He uses a mix of actual data, inaccurate data, and fabricated data to support this hypothesis. A logical consequence of his argument is that a reduction in the availability of AP courses will lead to an improvement in AP quality. In this paper, we maintain that his thesis is flawed because he confounds quality with scarcity. In contrast to his narrow conception of quality, quality in the AP context is subject- specific and multifaceted, embracing course content, the teacher, the student as well as the exam. Increased access will not diminish quality. Instead, increased access exposes students to college-level course material, encourages teachers to expand their knowledge domains, serves as a lever for lifting curriculum rigor, and provides students with the opportunity to experience the challenges associated with advanced placement in college

Advanced Placement: Access Not Exclusion

Lichten (2000) argues that increased access to AP courses in high schools has led to a decline in AP quality. He uses a mix of actual data, inaccurate data, and fabricated data to support this hypothesis. A logical consequence of his argument is that a reduction in the availability of AP courses will lead to an improvement in AP quality. In this paper, we maintain that his thesis is flawed because he confounds quality with scarcity. In contrast to his narrow conception of quality, quality in the AP context is subject- specific and multifaceted, embracing course content, the teacher, the student as well as the exam. Increased access will not diminish quality. Instead, increased access exposes students to college-level course material, encourages teachers to expand their knowledge domains, serves as a lever for lifting curriculum rigor, and provides students with the opportunity to experience the challenges associated with advanced placement in college