Adversarial training (AT) methods are effective against adversarial attacks,
yet they introduce severe disparity of accuracy and robustness between
different classes, known as the robust fairness problem. Previously proposed
Fair Robust Learning (FRL) adaptively reweights different classes to improve
fairness. However, the performance of the better-performed classes decreases,
leading to a strong performance drop. In this paper, we observed two unfair
phenomena during adversarial training: different difficulties in generating
adversarial examples from each class (source-class fairness) and disparate
target class tendencies when generating adversarial examples (target-class
fairness). From the observations, we propose Balance Adversarial Training (BAT)
to address the robust fairness problem. Regarding source-class fairness, we
adjust the attack strength and difficulties of each class to generate samples
near the decision boundary for easier and fairer model learning; considering
target-class fairness, by introducing a uniform distribution constraint, we
encourage the adversarial example generation process for each class with a fair
tendency. Extensive experiments conducted on multiple datasets (CIFAR-10,
CIFAR-100, and ImageNette) demonstrate that our method can significantly
outperform other baselines in mitigating the robust fairness problem (+5-10\%
on the worst class accuracy