A surge of interest has emerged in utilizing Transformers in diverse vision
tasks owing to its formidable performance. However, existing approaches
primarily focus on optimizing internal model architecture designs that often
entail significant trial and error with high burdens. In this work, we propose
a new paradigm dubbed Decision Stream Calibration that boosts the performance
of general Vision Transformers. To achieve this, we shed light on the
information propagation mechanism in the learning procedure by exploring the
correlation between different tokens and the relevance coefficient of multiple
dimensions. Upon further analysis, it was discovered that 1) the final decision
is associated with tokens of foreground targets, while token features of
foreground target will be transmitted into the next layer as much as possible,
and the useless token features of background area will be eliminated gradually
in the forward propagation. 2) Each category is solely associated with specific
sparse dimensions in the tokens. Based on the discoveries mentioned above, we
designed a two-stage calibration scheme, namely ViT-Calibrator, including token
propagation calibration stage and dimension propagation calibration stage.
Extensive experiments on commonly used datasets show that the proposed approach
can achieve promising results. The source codes are given in the supplements.Comment: 14pages, 12 figure