We present a novel adversarial penalized self-knowledge distillation method,
named adversarial learning and implicit regularization for self-knowledge
distillation (AI-KD), which regularizes the training procedure by adversarial
learning and implicit distillations. Our model not only distills the
deterministic and progressive knowledge which are from the pre-trained and
previous epoch predictive probabilities but also transfers the knowledge of the
deterministic predictive distributions using adversarial learning. The
motivation is that the self-knowledge distillation methods regularize the
predictive probabilities with soft targets, but the exact distributions may be
hard to predict. Our method deploys a discriminator to distinguish the
distributions between the pre-trained and student models while the student
model is trained to fool the discriminator in the trained procedure. Thus, the
student model not only can learn the pre-trained model's predictive
probabilities but also align the distributions between the pre-trained and
student models. We demonstrate the effectiveness of the proposed method with
network architectures on multiple datasets and show the proposed method
achieves better performance than state-of-the-art methods.Comment: 12 pages, 7 figure