DisWOT: Student Architecture Search for Distillation WithOut Training

Knowledge distillation (KD) is an effective training strategy to improve the lightweight student models under the guidance of cumbersome teachers. However, the large architecture difference across the teacher-student pairs limits the distillation gains. In contrast to previous adaptive distillation methods to reduce the teacher-student gap, we explore a novel training-free framework to search for the best student architectures for a given teacher. Our work first empirically show that the optimal model under vanilla training cannot be the winner in distillation. Secondly, we find that the similarity of feature semantics and sample relations between random-initialized teacher-student networks have good correlations with final distillation performances. Thus, we efficiently measure similarity matrixs conditioned on the semantic activation maps to select the optimal student via an evolutionary algorithm without any training. In this way, our student architecture search for Distillation WithOut Training (DisWOT) significantly improves the performance of the model in the distillation stage with at least 180$\times$ training acceleration. Additionally, we extend similarity metrics in DisWOT as new distillers and KD-based zero-proxies. Our experiments on CIFAR, ImageNet and NAS-Bench-201 demonstrate that our technique achieves state-of-the-art results on different search spaces. Our project and code are available at https://lilujunai.github.io/DisWOT-CVPR2023/.Comment: Accepted by CVPR202

OVO: One-shot Vision Transformer Search with Online distillation

Pure transformers have shown great potential for vision tasks recently. However, their accuracy in small or medium datasets is not satisfactory. Although some existing methods introduce a CNN as a teacher to guide the training process by distillation, the gap between teacher and student networks would lead to sub-optimal performance. In this work, we propose a new One-shot Vision transformer search framework with Online distillation, namely OVO. OVO samples sub-nets for both teacher and student networks for better distillation results. Benefiting from the online distillation, thousands of subnets in the supernet are well-trained without extra finetuning or retraining. In experiments, OVO-Ti achieves 73.32% top-1 accuracy on ImageNet and 75.2% on CIFAR-100, respectively.Comment: arXiv admin note: substantial text overlap with arXiv:2107.00651 by other author

ConvFormer: Closing the Gap Between CNN and Vision Transformers

Vision transformers have shown excellent performance in computer vision tasks. However, the computation cost of their (local) self-attention mechanism is expensive. Comparatively, CNN is more efficient with built-in inductive bias. Recent works show that CNN is promising to compete with vision transformers by learning their architecture design and training protocols. Nevertheless, existing methods either ignore multi-level features or lack dynamic prosperity, leading to sub-optimal performance. In this paper, we propose a novel attention mechanism named MCA, which captures different patterns of input images by multiple kernel sizes and enables input-adaptive weights with a gating mechanism. Based on MCA, we present a neural network named ConvFormer. ConvFormer adopts the general architecture of vision transformers, while replacing the (local) self-attention mechanism with our proposed MCA. Extensive experimental results demonstrated that ConvFormer achieves state-of-the-art performance on ImageNet classification, which outperforms similar-sized vision transformers(ViTs) and convolutional neural networks (CNNs). Moreover, for object detection on COCO and semantic segmentation tasks on ADE20K, ConvFormer also shows excellent performance compared with recently advanced methods. Code and models will be available

Estimation of Total Body Skeletal Muscle Mass in Chinese Adults: Prediction Model by Dual-Energy X-Ray Absorptiometry

Background: There are few reports on total body skeletal muscle mass (SM) in Chinese. The objective of this study is to establish a prediction model of SM for Chinese adults. Methodology: Appendicular lean soft tissue (ALST) was measured by dual energy X-ray absorptiometry (DXA) and SM by magnetic resonance image (MRI) in 66 Chinese adults (52 men and 14 women). Images of MRI were segmented into compartments including intermuscular adipose tissue (IMAT) and IMAT-free SM. Regression was used to fit the prediction model SM = c + k × ALST. Age and gender were adjusted in the fitted model. The piece-wise linear function was performed to further explore the effect of age on SM. ‘Leave-One-Out Cross Validation’ was utilized to evaluate the prediction performance. The significance of observed differences between predicted and actual SM was tested by t test and the level of agreement was assessed by the method of Bland and Altman. Results: Men had greater ALST and IMAT-free SM than women. ALST was the primary predictor and highly correlated with IMAT-free SM (R2 = 0.94, SEE = 1.11 kg, P<0.001). Age was an additional predictor (SM prediction model with age adjusted R2 = 0.95, SEE = 1.05 kg, P<0.001). There was a piece-wise linear relationship between age and IMAT-free SM: IMAT-free SM = 1.21×ALST−0.98, (Age <45 years) and IMAT-free SM = 1.21×ALST−0.98−0.04× (Age−45), (Age ≥45years). The prediction performance of this age-adjusted model was good due to ‘Leave-One-Out Cross Validation’. No significant difference between measured and predicted IMAT-free SM was detected. Conclusion: Previous SM prediction model developed in multi-ethnic groups underestimated SM by 2.3% and 3.4% for Chinese men and women. A new prediction model by DXA has been established to predict SM in Chinese adults

EMQ: Evolving Training-free Proxies for Automated Mixed Precision Quantization

Mixed-Precision Quantization~(MQ) can achieve a competitive accuracy-complexity trade-off for models. Conventional training-based search methods require time-consuming candidate training to search optimized per-layer bit-width configurations in MQ. Recently, some training-free approaches have presented various MQ proxies and significantly improve search efficiency. However, the correlation between these proxies and quantization accuracy is poorly understood. To address the gap, we first build the MQ-Bench-101, which involves different bit configurations and quantization results. Then, we observe that the existing training-free proxies perform weak correlations on the MQ-Bench-101. To efficiently seek superior proxies, we develop an automatic search of proxies framework for MQ via evolving algorithms. In particular, we devise an elaborate search space involving the existing proxies and perform an evolution search to discover the best correlated MQ proxy. We proposed a diversity-prompting selection strategy and compatibility screening protocol to avoid premature convergence and improve search efficiency. In this way, our Evolving proxies for Mixed-precision Quantization~(EMQ) framework allows the auto-generation of proxies without heavy tuning and expert knowledge. Extensive experiments on ImageNet with various ResNet and MobileNet families demonstrate that our EMQ obtains superior performance than state-of-the-art mixed-precision methods at a significantly reduced cost. The code will be released.Comment: Accepted by ICCV202

Adipose Tissue Quantification by Imaging Methods: A Proposed Classification

Recent advances in imaging techniques and understanding of differences in the molecular biology of adipose tissue has rendered classical anatomy obsolete, requiring a new classification of the topography of adipose tissue. Adipose tissue is one of the largest body compartments, yet a classification that defines specific adipose tissue depots based on their anatomic location and related functions is lacking. The absence of an accepted taxonomy poses problems for investigators studying adipose tissue topography and its functional correlates. The aim of this review was to critically examine the literature on imaging of whole body and regional adipose tissue and to create the first systematic classification of adipose tissue topography. Adipose tissue terminology was examined in over 100 original publications. Our analysis revealed inconsistencies in the use of specific definitions, especially for the compartment termed “visceral” adipose tissue. This analysis leads us to propose an updated classification of total body and regional adipose tissue, providing a well-defined basis for correlating imaging studies of specific adipose tissue depots with molecular processes

Lower limb skeletal muscle mass : development of dual-energy X-ray absorptiometry prediction model

Développement d'une méthode de mesure de la masse musculaire des membres inférieurs par absorptiométrie biophotonique. Cette méthode innovante est comparée à celle traditionnellement utilisée (Imagerie par résonance magnétique). Après comparaison des mesures, cette méthode est validé

Untargeted Metabolomics Analysis Revealed the Major Metabolites in the Seeds of four Polygonatum Species

Most Polygonatum species are widely used in China as a source of medicine and food. In this study, a UPLC-QTOF-MS/MS system was used to conduct an untargeted metabolomics analysis and compare the classes and relative contents of metabolites in the seeds of four Polygonatum species: P. sibiricum (Ps), P. cyrtonema (Pc), P. kingianum (Pk), and P. macropodium (Pm). The objectives of this study were to clarify the metabolic profiles of these seeds and to verify their medicinal and nutritional value via comparative analyses. A total of 873 metabolites were identified, including 185 flavonoids, 127 lipids, 105 phenolic acids, and 36 steroids. The comparative analysis of metabolites among Polygonatum seed samples indicated that flavonoids, steroids, and terpenoids were the main differentially abundant compounds. The results of principal component analysis and hierarchical clustering were consistent indicating that the metabolites in Ps and Pm are similar, but differ greatly from Pc and Pk. The data generated in this study provide additional evidence of the utility of Polygonatum seeds for producing food and medicine

Developed equations for predicting total body skeletal muscle mass.

<p>Values are means ± SD.</p><p><b>Abbreviations</b>: ALST, appendicular lean soft tissue; IMAT, intermuscular adipose tissue; SM, total body skeletal muscle.</p