Adversarial training (AT) formulated as the minimax optimization problem can
effectively enhance the model's robustness against adversarial attacks. The
existing AT methods mainly focused on manipulating the inner maximization for
generating quality adversarial variants or manipulating the outer minimization
for designing effective learning objectives. However, empirical results of AT
always exhibit the robustness at odds with accuracy and the existence of the
cross-over mixture problem, which motivates us to study some label randomness
for benefiting the AT. First, we thoroughly investigate noisy labels (NLs)
injection into AT's inner maximization and outer minimization, respectively and
obtain the observations on when NL injection benefits AT. Second, based on the
observations, we propose a simple but effective method -- NoiLIn that randomly
injects NLs into training data at each training epoch and dynamically increases
the NL injection rate once robust overfitting occurs. Empirically, NoiLIn can
significantly mitigate the AT's undesirable issue of robust overfitting and
even further improve the generalization of the state-of-the-art AT methods.
Philosophically, NoiLIn sheds light on a new perspective of learning with NLs:
NLs should not always be deemed detrimental, and even in the absence of NLs in
the training set, we may consider injecting them deliberately. Codes are
available in https://github.com/zjfheart/NoiLIn.Comment: Accepted at Transactions on Machine Learning Research (TMLR) at June
202