This paper proposes Mutual Information Regularized Assignment (MIRA), a
pseudo-labeling algorithm for unsupervised representation learning inspired by
information maximization. We formulate online pseudo-labeling as an
optimization problem to find pseudo-labels that maximize the mutual information
between the label and data while being close to a given model probability. We
derive a fixed-point iteration method and prove its convergence to the optimal
solution. In contrast to baselines, MIRA combined with pseudo-label prediction
enables a simple yet effective clustering-based representation learning without
incorporating extra training techniques or artificial constraints such as
sampling strategy, equipartition constraints, etc. With relatively small
training epochs, representation learned by MIRA achieves state-of-the-art
performance on various downstream tasks, including the linear/k-NN evaluation
and transfer learning. Especially, with only 400 epochs, our method applied to
ImageNet dataset with ResNet-50 architecture achieves 75.6% linear evaluation
accuracy.Comment: NeurIPS 202