Ensembled CTR Prediction via Knowledge Distillation

Recently, deep learning-based models have been widely studied for click-through rate (CTR) prediction and lead to improved prediction accuracy in many industrial applications. However, current research focuses primarily on building complex network architectures to better capture sophisticated feature interactions and dynamic user behaviors. The increased model complexity may slow down online inference and hinder its adoption in real-time applications. Instead, our work targets at a new model training strategy based on knowledge distillation (KD). KD is a teacher-student learning framework to transfer knowledge learned from a teacher model to a student model. The KD strategy not only allows us to simplify the student model as a vanilla DNN model but also achieves significant accuracy improvements over the state-of-the-art teacher models. The benefits thus motivate us to further explore the use of a powerful ensemble of teachers for more accurate student model training. We also propose some novel techniques to facilitate ensembled CTR prediction, including teacher gating and early stopping by distillation loss. We conduct comprehensive experiments against 12 existing models and across three industrial datasets. Both offline and online A/B testing results show the effectiveness of our KD-based training strategy.Comment: Published in CIKM'202

Modeling the spatiotemporal dynamics of industrial sulfur dioxide emissions in China based on DMSP-OLS nighttime stable light data.

Due to the rapid economic growth and the heavy reliance on fossil fuels, China has become one of the countries with the highest sulfur dioxide (SO2) emissions, which pose a severe challenge to human health and the sustainable development of social economy. In order to cope with the serious problem of SO2 pollution, this study attempts to explore the spatial temporal variations of industrial SO2 emissions in China utilizing the Defense Meteorological Satellite Program's Operational Linescan System (DMSP-OLS) nighttime stable light (NSL) data. We first explored the relationship between the NSL data and the statistical industrial SO2 emissions at the provincial level, and confirmed that there was a positive correlation between these two datasets. Consequently, 17 linear regression models were established based on the NSL data and the provincial statistical emissions to model the spatial-temporal dynamics of China's industrial SO2 emissions from 1997 to 2013. Next, the NSL-based estimated results were evaluated utilizing the prefectural statistical industrial SO2 emissions and emission inventory data, respectively. Finally, the distribution of China's industrial SO2 emissions at 1 km spatial resolution were estimated, and the temporal and spatial dynamics were explored from multiple scales (national scale, regional scale and scale of urban agglomeration). The results show that: (1) The NSL data can be successfully applied to estimate the dynamic changes of China's industrial SO2 emissions. The coefficient of determination (R2) values of the NSL-based estimation results in most years were greater than 0.6, and the relative error (RE) values were less than 10%, when validated by the prefectural statistical SO2 emissions. Moreover, compared with the inventory emissions, the adjusted coefficient of determination (Adj.R-Square) reached 0.61, with the significance at the 0.001 level. (2) During the observation period, the temporal and spatial dynamics of industrial SO2 emissions varied greatly in different regions. The high growth type was largely distributed in China's Western region, Central region, and Shandong Peninsula, while the no-obvious-growth type was concentrated in Western region, Beijing-Tianjin-Tangshan and Middle south of Liaoning. The high grade of industrial SO2 emissions was mostly concentrated in China's Eastern region, Western region, Shanghai-Nanjing-Hangzhou and Shandong Peninsula, while the low grade mainly concentrated in China's Western region, Middle south of Liaoning and Beijing-Tianjin-Tangshan. These results of our research can not only enhance the understanding of the spatial-temporal dynamics of industrial SO2 emissions in China, but also offer some scientific references for formulating feasible industrial SO2 emission reduction policies