Today, algorithmic models are shaping important decisions in domains such as
credit, employment, or criminal justice. At the same time, these algorithms
have been shown to have discriminatory effects. Some organizations have tried
to mitigate these effects by removing demographic features from an algorithm's
inputs. If an algorithm is not provided with a feature, one might think, then
its outputs should not discriminate with respect to that feature. This may not
be true, however, when there are other correlated features. In this paper, we
explore the limits of this approach using a unique opportunity created by a
lawsuit settlement concerning discrimination on Facebook's advertising
platform. Facebook agreed to modify its Lookalike Audiences tool - which
creates target sets of users for ads by identifying users who share "common
qualities" with users in a source audience provided by an advertiser - by
removing certain demographic features as inputs to its algorithm. The modified
tool, Special Ad Audiences, is intended to reduce the potential for
discrimination in target audiences. We create a series of Lookalike and Special
Ad audiences based on biased source audiences - i.e., source audiences that
have known skew along the lines of gender, age, race, and political leanings.
We show that the resulting Lookalike and Special Ad audiences both reflect
these biases, despite the fact that Special Ad Audiences algorithm is not
provided with the features along which our source audiences are skewed. More
broadly, we provide experimental proof that removing demographic features from
a real-world algorithmic system's inputs can fail to prevent biased outputs.
Organizations using algorithms to mediate access to life opportunities should
consider other approaches to mitigating discriminatory effects