Decoupled Knowledge Distillation

State-of-the-art distillation methods are mainly based on distilling deep features from intermediate layers, while the significance of logit distillation is greatly overlooked. To provide a novel viewpoint to study logit distillation, we reformulate the classical KD loss into two parts, i.e., target class knowledge distillation (TCKD) and non-target class knowledge distillation (NCKD). We empirically investigate and prove the effects of the two parts: TCKD transfers knowledge concerning the "difficulty" of training samples, while NCKD is the prominent reason why logit distillation works. More importantly, we reveal that the classical KD loss is a coupled formulation, which (1) suppresses the effectiveness of NCKD and (2) limits the flexibility to balance these two parts. To address these issues, we present Decoupled Knowledge Distillation (DKD), enabling TCKD and NCKD to play their roles more efficiently and flexibly. Compared with complex feature-based methods, our DKD achieves comparable or even better results and has better training efficiency on CIFAR-100, ImageNet, and MS-COCO datasets for image classification and object detection tasks. This paper proves the great potential of logit distillation, and we hope it will be helpful for future research. The code is available at https://github.com/megvii-research/mdistiller.Comment: Accepted by CVPR2022, fix typ

A simplified regression building thermal model of detached two-floor house in U.S. for virtual energy storage control

The residential building sector accounts for approximately 37% of total U.S. electricity consumption. Within the residential building sector, heating and cooling is the main target for peak load shifting/reduction since it is the largest contributor to peak demand. In fact, the flexibility of residential HVAC loads can provide continuous variation of demand to provide grid services. Such load can be taken as virtual energy storage (VES) resources by varying their demand over a baseline so that they appear to be providing the same service as electrical battery energy storage to the grid. The performance of HVAC load control to provide VES relies heavily on the accuracy of indoor temperature or cooling/heating demand predictions and therefore the quality of building model. Most building models can be broken down into two categories: gray-box models, i.e. Resistance-Capacitance (RC) model and data-mining based models. The RC model, built based on a combination of physical principles and measured data, is constituted with electrical analogue pattern with resistance (R) and capacitance (C). Different RC models with different structures and orders have been widely applied in prediction of building indoor temperature or cooling/heating demand. But, in general, the RC models require long periods of data to train model coefficients as well as considerable training time and computation burden. Recently, data-mining based models have gained increasing interest due to their capability in analyzing large-scale data and flexibility in practical applications. But, the data-mining based models tend to have invisible model structures which poses a problem when trying to use the model for optimal control or model predictive control of the HVAC system. Hence, for this control-oriented modeling scope, there is a continuing need for efficient online system identification techniques, which can provide explicit parameters for the model. Traditional regression models fit well for this specific purpose. This paper presents a rolling horizon building thermal model for identifying indoor temperatures in separate floors of a typical detached residential house. The developed model includes online adaptive correction component. Since this adaptive algorithm needs to be implemented online, a less computation-demanding polynomial fitting algorithm is adopted. This kind of fitting problem can be cast as linear regression problem with multiple variables, parameters of which can be efficiently obtained by well-known gradient descent method. The validation is realized by comparing the predicted results with the measured data from a real typical detached two-floor house. The results show that the developed model has satisfactory performance in predicting the building indoor temperature in 1st and 2nd floors

Machine-Learning Model of Electric Water Heater for Electricity Consumption Prediction

The recent increase of smart meters in the residential sector has led to large available datasets. The electricity consumption of individual households/devices can be accessed in close to real time, and allows both the demand and supply side to extract valuable information for efficient energy management. Predicting electricity consumption should help utilities improve planning generation and demand side management, however this is not a trivial task as consumption at the individual household level varies with occupant behavior. In residential buildings, many loads have some power flexibility. One of them is water heater, which accounts for up to 20% of home daily electricity use. Conventional methods for water heater power prediction, which heavily rely on physical principles, have limited applicability as their performance is subject to many physical assumptions. Recently, black-box models have gained huge interest due to their flexibility in model development and the rich availability of data in modern buildings. Black-box modelling methods can be further categorized into two types, i.e., statistical methods and supervised machine learning (ML). While statistical methods are relatively easy to implement, they can only capture linear relationships among building variables. Since building operations are typically complicated and nonlinear, the resulting accuracy can be poor. Many ML-based, data-driven approaches have the ability to characterize and forecast total energy consumption of commercial data. However, a paucity of research applying data-driven methods have been tested on the hour ahead energy consumption forecasts for typical detached residential houses in the US. With the advances in smart metering, sub meter usage forecasts at the household-level is also gaining popularity for smart building control and demand response programs. This led us to develop a hybrid model to address the problem of residential hour and day ahead load forecasting through the integration of data-driven techniques. The developed forecasting models are built using three common ML algorithms, support vector machines(SVM), Gaussian Naire Bayes, and Random Forest. Performance comparison among these ML methods was carried out. The results suggest that all models were able to correctly predict a greater proportion of the actual power consumption with prediction accuracy yields between 94% ~ 96%. The SVM model performs the best, while the RF works the worst

Shape Memory Alloy Helical Microrobots with Transformable Capability towards Vascular Occlusion Treatment

Practical implementation of minimally invasive biomedical applications has been a long-sought goal for microrobots. In this field, most previous studies only demonstrate microrobots with locomotion ability or performing a single task, unable to be functionalized effectively. Here, we propose a biocompatible shape memory alloy helical microrobot with regulative structure transformation, making it possible to adjust its motion behavior and mechanical properties precisely. Especially, towards vascular occlusion problem, these microrobots reveal a fundamental solution strategy in the mechanical capability using shape memory effect. Such shape-transformable microrobots can not only manipulate thrust and torque by structure to enhance the unclogging efficiency as a microdriller but also utilize the high work energy to apply the expandable helical tail as a self-propulsive stent. The strategy takes advantage of untethered manipulation to operate microsurgery without unnecessary damage. This study opens a route to functionalize microrobots via accurate tuning in structures, motions, and mechanical properties

Evaluating the Adaptability of Reinforcement Learning Based HVAC Control for Residential Houses

Intelligent Heating, Ventilation, and Air Conditioning (HVAC) control using deep reinforcement learning (DRL) has recently gained a lot of attention due to its ability to optimally control the complex behavior of the HVAC system. However, more exploration is needed on understanding the adaptability challenges that the DRL agent could face during the deployment phase. Using online learning for such applications is not realistic due to the long learning period and likely poor comfort control during the learning process. Alternatively, DRL can be pre-trained using a building model prior to deployment. However, developing an accurate building model for every house and deploying a pre-trained DRL model for HVAC control would not be cost-effective. In this study, we focus on evaluating the ability of DRL-based HVAC control to provide cost savings when pre-trained on one building model and deployed on different house models with varying user comforts. We observed around 30% of cost reduction by pre-trained model over baseline when validated in a simulation environment and achieved up to 21% cost reduction when deployed in the real house. This finding provides experimental evidence that the pre-trained DRL has the potential to adapt to different house environments and comfort settings