DisWOT: Student Architecture Search for Distillation WithOut Training

Knowledge distillation (KD) is an effective training strategy to improve the lightweight student models under the guidance of cumbersome teachers. However, the large architecture difference across the teacher-student pairs limits the distillation gains. In contrast to previous adaptive distillation methods to reduce the teacher-student gap, we explore a novel training-free framework to search for the best student architectures for a given teacher. Our work first empirically show that the optimal model under vanilla training cannot be the winner in distillation. Secondly, we find that the similarity of feature semantics and sample relations between random-initialized teacher-student networks have good correlations with final distillation performances. Thus, we efficiently measure similarity matrixs conditioned on the semantic activation maps to select the optimal student via an evolutionary algorithm without any training. In this way, our student architecture search for Distillation WithOut Training (DisWOT) significantly improves the performance of the model in the distillation stage with at least 180$\times$ training acceleration. Additionally, we extend similarity metrics in DisWOT as new distillers and KD-based zero-proxies. Our experiments on CIFAR, ImageNet and NAS-Bench-201 demonstrate that our technique achieves state-of-the-art results on different search spaces. Our project and code are available at https://lilujunai.github.io/DisWOT-CVPR2023/.Comment: Accepted by CVPR202

NORM: Knowledge Distillation via N-to-One Representation Matching

Existing feature distillation methods commonly adopt the One-to-one Representation Matching between any pre-selected teacher-student layer pair. In this paper, we present N-to-One Representation (NORM), a new two-stage knowledge distillation method, which relies on a simple Feature Transform (FT) module consisting of two linear layers. In view of preserving the intact information learnt by the teacher network, during training, our FT module is merely inserted after the last convolutional layer of the student network. The first linear layer projects the student representation to a feature space having N times feature channels than the teacher representation from the last convolutional layer, and the second linear layer contracts the expanded output back to the original feature space. By sequentially splitting the expanded student representation into N non-overlapping feature segments having the same number of feature channels as the teacher's, they can be readily forced to approximate the intact teacher representation simultaneously, formulating a novel many-to-one representation matching mechanism conditioned on a single teacher-student layer pair. After training, such an FT module will be naturally merged into the subsequent fully connected layer thanks to its linear property, introducing no extra parameters or architectural modifications to the student network at inference. Extensive experiments on different visual recognition benchmarks demonstrate the leading performance of our method. For instance, the ResNet18|MobileNet|ResNet50-1/4 model trained by NORM reaches 72.14%|74.26%|68.03% top-1 accuracy on the ImageNet dataset when using a pre-trained ResNet34|ResNet50|ResNet50 model as the teacher, achieving an absolute improvement of 2.01%|4.63%|3.03% against the individually trained counterpart. Code is available at https://github.com/OSVAI/NORMComment: The paper of NORM is published at ICLR 2023. Code and models are available at https://github.com/OSVAI/NOR

Facile synthesis of superhydrophobic surface of ZnO nanoflakes: chemical coating and UV-induced wettability conversion

This work reports an oriented growth process of two-dimensional (2D) ZnO nanoflakes on aluminum substrate through a low temperature hydrothermal technique and proposes the preliminary growth mechanism. A bionic superhydrophobic surface with excellent corrosion protection over a wide pH range in both acidic and alkaline solutions was constructed by a chemical coating treatment with stearic acid (SA) molecules on ZnO nanoflakes. It is found that the superhydrophobic surface of ZnO nanoflake arrays shows a maximum water contact angle (CA) of 157° and a low sliding angle of 8°, and it can be reversibly switched to its initial superhydrophilic state under ultraviolet (UV) irradiation, which is due to the UV-induced decomposition of the coated SA molecules. This study is significant for simple and inexpensive building of large-scale 2D ZnO nanoflake arrays with special wettability which can extend the applications of ZnO films to many other important fields

Morphology-dependent field emission properties and wetting behavior of ZnO nanowire arrays

The fabrication of three kinds of ZnO nanowire arrays with different structural parameters over Au-coated silicon (100) by facile thermal evaporation of ZnS precursor is reported, and the growth mechanism are proposed based on structural analysis. Field emission (FE) properties and wetting behavior were revealed to be strongly morphology dependent. The nanowire arrays in small diameter and high aspect ratio exhibited the best FE performance showing a low turn-on field (4.1 V/μm) and a high field-enhancement factor (1745.8). The result also confirmed that keeping large air within the films was an effective way to obtain super water-repellent properties. This study indicates that the preparation of ZnO nanowire arrays in an optimum structural model is crucial to FE efficiency and wetting behavior

Characteristics of bacterial communities in shallow and thin heavy oil reservoir

Revealing the characteristics of microorganisms that inhabit oil reservoirs is important in the effective application of microbial enhanced oil recovery (MEOR) technique. Plenty of studies have been conducted to discover microbial communities in light oil reservoirs, but investigations on the characteristics of bacterial communities in shallow and thin heavy oil reservoirs are limited. The aim of this study is to investigate bacterial communities in shallow and thin heavy oil reservoir, an oilfield in Henan (China) was taken as an example, and the 16S rDNA clone library approach was adopted to analyze the composition, abundance, and distribution of bacterial communities. A total of 682 sequences obtained from the four clone libraries were assigned to 84 operational taxonomic units (OTU) and 11 bacterial groups were identified in the oil reservoir. Results demonstrate the following: (1) The heavy oil reservoir has low bacterial diversity. (2) Differences exist in the bacterial community structures of the clone libraries. (3) The distribution of bacterial communities is consistent with the temperature, salinity, and oil properties of the oil reservoir. The findings of this study can provide basic theoretical guidance for the application of MEOR in shallow and thin heavy oil reservoirs.</p

The moderate level of digital transformation: from the perspective of green total factor productivity

In the context of accelerated development of the digital economy, whether enterprises can drive green total factor productivity (GTFP) through digital technology has become the key to promoting high-quality development of the economy and achieving the goal of "dual-carbon", However, the relationship between digital transformation and GTFP is still controversial in existing studies. Based on the data of 150 listed companies in China's A-share energy industry from 2011 to 2021, this study empirically analyzes the impact of digital transformation on GTFP using a fixed-effect model. The study shows an inverted U-shaped nonlinear effect of digital transformation on enterprises' GTFP, and the conclusion still holds after a series of robustness tests. Mechanism analysis shows that enterprise investment efficiency and labour allocation efficiency play a significant mediating role in the above inverted U-shaped relationship, in which the inverted U-shaped relationship between digital transformation and GTFP mainly stems from the influence of enterprise investment efficiency. Heterogeneity analysis finds that the inverted U-shaped relationship between digital transformation and GTFP of enterprises is more significant in large-scale enterprises, new energy enterprises and enterprises in central and western regions. The study's findings provide important insights for enterprises to promote digital transformation and realize the green and high-quality development of the energy industry

CTC-Segmentation of Large Corpora for German End-to-end Speech Recognition

Recent end-to-end Automatic Speech Recognition (ASR) systems demonstrated the ability to outperform conventional hybrid DNN/ HMM ASR. Aside from architectural improvements in those systems, those models grew in terms of depth, parameters and model capacity. However, these models also require more training data to achieve comparable performance. In this work, we combine freely available corpora for German speech recognition, including yet unlabeled speech data, to a big dataset of over $1700$h of speech data. For data preparation, we propose a two-stage approach that uses an ASR model pre-trained with Connectionist Temporal Classification (CTC) to boot-strap more training data from unsegmented or unlabeled training data. Utterances are then extracted from label probabilities obtained from the network trained with CTC to determine segment alignments. With this training data, we trained a hybrid CTC/attention Transformer model that achieves $12.8\%$ WER on the Tuda-DE test set, surpassing the previous baseline of $14.4\%$ of conventional hybrid DNN/HMM ASR.Comment: Published at SPECOM 202