Transfer-based adversarial attacks can effectively evaluate model robustness
in the black-box setting. Though several methods have demonstrated impressive
transferability of untargeted adversarial examples, targeted adversarial
transferability is still challenging. The existing methods either have low
targeted transferability or sacrifice computational efficiency. In this paper,
we develop a simple yet practical framework to efficiently craft targeted
transfer-based adversarial examples. Specifically, we propose a conditional
generative attacking model, which can generate the adversarial examples
targeted at different classes by simply altering the class embedding and share
a single backbone. Extensive experiments demonstrate that our method improves
the success rates of targeted black-box attacks by a significant margin over
the existing methods -- it reaches an average success rate of 29.6\% against
six diverse models based only on one substitute white-box model in the standard
testing of NeurIPS 2017 competition, which outperforms the state-of-the-art
gradient-based attack methods (with an average success rate of <2\%) by a
large margin. Moreover, the proposed method is also more efficient beyond an
order of magnitude than gradient-based methods