Social media platforms curate access to information and opportunities, and so
play a critical role in shaping public discourse today. The opaque nature of
the algorithms these platforms use to curate content raises societal questions.
Prior studies have used black-box methods to show that these algorithms can
lead to biased or discriminatory outcomes. However, existing auditing methods
face fundamental limitations because they function independent of the
platforms. Concerns of potential harm have prompted proposal of legislation in
both the U.S. and the E.U. to mandate a new form of auditing where vetted
external researchers get privileged access to social media platforms.
Unfortunately, to date there have been no concrete technical proposals to
provide such auditing, because auditing at scale risks disclosure of users'
private data and platforms' proprietary algorithms. We propose a new method for
platform-supported auditing that can meet the goals of the proposed
legislation. Our first contribution is to enumerate the challenges of existing
auditing methods to implement these policies at scale. Second, we suggest that
limited, privileged access to relevance estimators is the key to enabling
generalizable platform-supported auditing by external researchers. Third, we
show platform-supported auditing need not risk user privacy nor disclosure of
platforms' business interests by proposing an auditing framework that protects
against these risks. For a particular fairness metric, we show that ensuring
privacy imposes only a small constant factor increase (6.34x as an upper bound,
and 4x for typical parameters) in the number of samples required for accurate
auditing. Our technical contributions, combined with ongoing legal and policy
efforts, can enable public oversight into how social media platforms affect
individuals and society by moving past the privacy-vs-transparency hurdle