To capture the relationship between samples and labels, conditional
generative models often inherit spurious correlations from the training
dataset. This can result in label-conditional distributions that are imbalanced
with respect to another latent attribute. To mitigate this issue, which we call
spurious causality of conditional generation, we propose a general two-step
strategy. (a) Fairness Intervention (FI): emphasize the minority samples that
are hard to generate due to the spurious correlation in the training dataset.
(b) Corrective Sampling (CS): explicitly filter the generated samples and
ensure that they follow the desired latent attribute distribution. We have
designed the fairness intervention to work for various degrees of supervision
on the spurious attribute, including unsupervised, weakly-supervised, and
semi-supervised scenarios. Our experimental results demonstrate that FICS can
effectively resolve spurious causality of conditional generation across various
datasets.Comment: TMLR 202