Medical imaging models have been shown to encode information about patient
demographics (age, race, sex) in their latent representation, raising concerns
about their potential for discrimination. Here, we ask whether it is feasible
and desirable to train models that do not encode demographic attributes. We
consider different types of invariance with respect to demographic attributes -
marginal, class-conditional, and counterfactual model invariance - and lay out
their equivalence to standard notions of algorithmic fairness. Drawing on
existing theory, we find that marginal and class-conditional invariance can be
considered overly restrictive approaches for achieving certain fairness
notions, resulting in significant predictive performance losses. Concerning
counterfactual model invariance, we note that defining medical image
counterfactuals with respect to demographic attributes is fraught with
complexities. Finally, we posit that demographic encoding may even be
considered advantageous if it enables learning a task-specific encoding of
demographic features that does not rely on human-constructed categories such as
'race' and 'gender'. We conclude that medical imaging models may need to encode
demographic attributes, lending further urgency to calls for comprehensive
model fairness assessments in terms of predictive performance