Less is More: Focus Attention for Efficient DETR

DETR-like models have significantly boosted the performance of detectors and even outperformed classical convolutional models. However, all tokens are treated equally without discrimination brings a redundant computational burden in the traditional encoder structure. The recent sparsification strategies exploit a subset of informative tokens to reduce attention complexity maintaining performance through the sparse encoder. But these methods tend to rely on unreliable model statistics. Moreover, simply reducing the token population hinders the detection performance to a large extent, limiting the application of these sparse models. We propose Focus-DETR, which focuses attention on more informative tokens for a better trade-off between computation efficiency and model accuracy. Specifically, we reconstruct the encoder with dual attention, which includes a token scoring mechanism that considers both localization and category semantic information of the objects from multi-scale feature maps. We efficiently abandon the background queries and enhance the semantic interaction of the fine-grained object queries based on the scores. Compared with the state-of-the-art sparse DETR-like detectors under the same setting, our Focus-DETR gets comparable complexity while achieving 50.4AP (+2.2) on COCO. The code is available at https://github.com/huawei-noah/noah-research/tree/master/Focus-DETR and https://gitee.com/mindspore/models/tree/master/research/cv/Focus-DETR.Comment: 8 pages, 6 figures, accepted to ICCV202

One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation

Knowledge distillation~(KD) has proven to be a highly effective approach for enhancing model performance through a teacher-student training scheme. However, most existing distillation methods are designed under the assumption that the teacher and student models belong to the same model family, particularly the hint-based approaches. By using centered kernel alignment (CKA) to compare the learned features between heterogeneous teacher and student models, we observe significant feature divergence. This divergence illustrates the ineffectiveness of previous hint-based methods in cross-architecture distillation. To tackle the challenge in distilling heterogeneous models, we propose a simple yet effective one-for-all KD framework called OFA-KD, which significantly improves the distillation performance between heterogeneous architectures. Specifically, we project intermediate features into an aligned latent space such as the logits space, where architecture-specific information is discarded. Additionally, we introduce an adaptive target enhancement scheme to prevent the student from being disturbed by irrelevant information. Extensive experiments with various architectures, including CNN, Transformer, and MLP, demonstrate the superiority of our OFA-KD framework in enabling distillation between heterogeneous architectures. Specifically, when equipped with our OFA-KD, the student models achieve notable performance improvements, with a maximum gain of 8.0% on the CIFAR-100 dataset and 0.7% on the ImageNet-1K dataset. PyTorch code and checkpoints can be found at https://github.com/Hao840/OFAKD

Data Upcycling Knowledge Distillation for Image Super-Resolution

Knowledge distillation (KD) emerges as a challenging yet promising technique for compressing deep learning models, characterized by the transmission of extensive learning representations from proficient and computationally intensive teacher models to compact student models. However, only a handful of studies have endeavored to compress the models for single image super-resolution (SISR) through KD, with their effects on student model enhancement remaining marginal. In this paper, we put forth an approach from the perspective of efficient data utilization, namely, the Data Upcycling Knowledge Distillation (DUKD) which facilitates the student model by the prior knowledge teacher provided via upcycled in-domain data derived from their inputs. This upcycling process is realized through two efficient image zooming operations and invertible data augmentations which introduce the label consistency regularization to the field of KD for SISR and substantially boosts student model's generalization. The DUKD, due to its versatility, can be applied across a broad spectrum of teacher-student architectures. Comprehensive experiments across diverse benchmarks demonstrate that our proposed DUKD method significantly outperforms previous art, exemplified by an increase of up to 0.5dB in PSNR over baselines methods, and a 67% parameters reduced RCAN model's performance remaining on par with that of the RCAN teacher model

Forecasting Regional Carbon Prices in China Based on Secondary Decomposition and a Hybrid Kernel-Based Extreme Learning Machine

Accurately forecasting carbon prices is key to managing associated risks in the financial market for carbon. To this end, the traditional strategy does not adequately decompose carbon prices, and the kernel extreme learning machine (KELM) with a single kernel function struggles to adapt to the nonlinearity, nonstationarity, and multiple frequencies of regional carbon prices in China. This study constructs a model, called the VMD-ICEEMDAN-RE-SSA-HKELM model, to forecast regional carbon prices in China based on the idea of ‘decomposition–reconstruction–integration’. The VMD is first used to decompose carbon prices and the ICEEMDAN is then used to decompose the residual term that contains complex information. To reduce the systematic error caused by increases in the mode components of carbon price, range entropy (RE) is used to reconstruct the results of its secondary decomposition. Following this, HKELM is optimized by the sparrow search algorithm and used to forecast each subseries of carbon prices. Finally, predictions of the price of carbon are obtained by linearly superimposing the results of the forecasts of each of its subseries. The results of experiments show that the secondary decomposition strategy proposed in this paper is superior to the traditional decomposition strategy, and the proposed model for forecasting carbon prices has significant advantages over a considered reference group of models

Forecasting Regional Carbon Prices in China Based on Secondary Decomposition and a Hybrid Kernel-Based Extreme Learning Machine

Accurately forecasting carbon prices is key to managing associated risks in the financial market for carbon. To this end, the traditional strategy does not adequately decompose carbon prices, and the kernel extreme learning machine (KELM) with a single kernel function struggles to adapt to the nonlinearity, nonstationarity, and multiple frequencies of regional carbon prices in China. This study constructs a model, called the VMD-ICEEMDAN-RE-SSA-HKELM model, to forecast regional carbon prices in China based on the idea of ‘decomposition–reconstruction–integration’. The VMD is first used to decompose carbon prices and the ICEEMDAN is then used to decompose the residual term that contains complex information. To reduce the systematic error caused by increases in the mode components of carbon price, range entropy (RE) is used to reconstruct the results of its secondary decomposition. Following this, HKELM is optimized by the sparrow search algorithm and used to forecast each subseries of carbon prices. Finally, predictions of the price of carbon are obtained by linearly superimposing the results of the forecasts of each of its subseries. The results of experiments show that the secondary decomposition strategy proposed in this paper is superior to the traditional decomposition strategy, and the proposed model for forecasting carbon prices has significant advantages over a considered reference group of models

Analysis of the REs of VMFs and IMFs.

Accurately predicting carbon price is crucial for risk avoidance in the carbon financial market. In light of the complex characteristics of the regional carbon price in China, this paper proposes a model to forecast carbon price based on the multi-factor hybrid kernel-based extreme learning machine (HKELM) by combining secondary decomposition and ensemble learning. Variational mode decomposition (VMD) is first used to decompose the carbon price into several modes, and range entropy is then used to reconstruct these modes. The multi-factor HKELM optimized by the sparrow search algorithm is used to forecast the reconstructed subsequences, where the main external factors innovatively selected by maximum information coefficient and historical time-series data on carbon prices are both considered as input variables to the forecasting model. Following this, the improved complete ensemble-based empirical mode decomposition with adaptive noise and range entropy are respectively used to decompose and reconstruct the residual term generated by VMD. Finally, the nonlinear ensemble learning method is introduced to determine the predictions of residual term and final carbon price. In the empirical analysis of Guangzhou market, the root mean square error(RMSE), mean absolute error (MAE) and mean absolute percentage error (MAPE) of the model are 0.1716, 0.1218 and 0.0026, respectively. The proposed model outperforms other comparative models in predicting accuracy. The work here extends the research on forecasting theory and methods of predicting the carbon price.</div

Implementation steps of ICEEMDAN.

Accurately predicting carbon price is crucial for risk avoidance in the carbon financial market. In light of the complex characteristics of the regional carbon price in China, this paper proposes a model to forecast carbon price based on the multi-factor hybrid kernel-based extreme learning machine (HKELM) by combining secondary decomposition and ensemble learning. Variational mode decomposition (VMD) is first used to decompose the carbon price into several modes, and range entropy is then used to reconstruct these modes. The multi-factor HKELM optimized by the sparrow search algorithm is used to forecast the reconstructed subsequences, where the main external factors innovatively selected by maximum information coefficient and historical time-series data on carbon prices are both considered as input variables to the forecasting model. Following this, the improved complete ensemble-based empirical mode decomposition with adaptive noise and range entropy are respectively used to decompose and reconstruct the residual term generated by VMD. Finally, the nonlinear ensemble learning method is introduced to determine the predictions of residual term and final carbon price. In the empirical analysis of Guangzhou market, the root mean square error(RMSE), mean absolute error (MAE) and mean absolute percentage error (MAPE) of the model are 0.1716, 0.1218 and 0.0026, respectively. The proposed model outperforms other comparative models in predicting accuracy. The work here extends the research on forecasting theory and methods of predicting the carbon price.</div

Results of correlation between external factors and carbon price in Hubei.

Results of correlation between external factors and carbon price in Hubei.</p

PACF results for the subseries of the carbon price.

PACF results for the subseries of the carbon price.</p