We devise deep nearest centroids (DNC), a conceptually elegant yet
surprisingly effective network for large-scale visual recognition, by
revisiting Nearest Centroids, one of the most classic and simple classifiers.
Current deep models learn the classifier in a fully parametric manner, ignoring
the latent data structure and lacking simplicity and explainability. DNC
instead conducts nonparametric, case-based reasoning; it utilizes sub-centroids
of training samples to describe class distributions and clearly explains the
classification as the proximity of test data and the class sub-centroids in the
feature space. Due to the distance-based nature, the network output
dimensionality is flexible, and all the learnable parameters are only for data
embedding. That means all the knowledge learnt for ImageNet classification can
be completely transferred for pixel recognition learning, under the
"pre-training and fine-tuning" paradigm. Apart from its nested simplicity and
intuitive decision-making mechanism, DNC can even possess ad-hoc explainability
when the sub-centroids are selected as actual training images that humans can
view and inspect. Compared with parametric counterparts, DNC performs better on
image classification (CIFAR-10, ImageNet) and greatly boots pixel recognition
(ADE20K, Cityscapes), with improved transparency and fewer learnable
parameters, using various network architectures (ResNet, Swin) and segmentation
models (FCN, DeepLabV3, Swin). We feel this work brings fundamental insights
into related fields.Comment: 23 pages, 8 figure