Building Variable-sized Models via Learngene Pool

Recently, Stitchable Neural Networks (SN-Net) is proposed to stitch some pre-trained networks for quickly building numerous networks with different complexity and performance trade-offs. In this way, the burdens of designing or training the variable-sized networks, which can be used in application scenarios with diverse resource constraints, are alleviated. However, SN-Net still faces a few challenges. 1) Stitching from multiple independently pre-trained anchors introduces high storage resource consumption. 2) SN-Net faces challenges to build smaller models for low resource constraints. 3). SN-Net uses an unlearned initialization method for stitch layers, limiting the final performance. To overcome these challenges, motivated by the recently proposed Learngene framework, we propose a novel method called Learngene Pool. Briefly, Learngene distills the critical knowledge from a large pre-trained model into a small part (termed as learngene) and then expands this small part into a few variable-sized models. In our proposed method, we distill one pretrained large model into multiple small models whose network blocks are used as learngene instances to construct the learngene pool. Since only one large model is used, we do not need to store more large models as SN-Net and after distilling, smaller learngene instances can be created to build small models to satisfy low resource constraints. We also insert learnable transformation matrices between the instances to stitch them into variable-sized models to improve the performance of these models. Exhaustive experiments have been implemented and the results validate the effectiveness of the proposed Learngene Pool compared with SN-Net

Transformer as Linear Expansion of Learngene

We propose expanding the shared Transformer module to produce and initialize Transformers of varying depths, enabling adaptation to diverse resource constraints. Drawing an analogy to genetic expansibility, we term such module as learngene. To identify the expansion mechanism, we delve into the relationship between the layer's position and its corresponding weight value, and find that linear function appropriately approximates this relationship. Building on this insight, we present Transformer as Linear Expansion of learnGene (TLEG), a novel approach for flexibly producing and initializing Transformers of diverse depths. Specifically, to learn learngene, we firstly construct an auxiliary Transformer linearly expanded from learngene, after which we train it through employing soft distillation. Subsequently, we can produce and initialize Transformers of varying depths via linearly expanding the well-trained learngene, thereby supporting diverse downstream scenarios. Extensive experiments on ImageNet-1K demonstrate that TLEG achieves comparable or better performance in contrast to many individual models trained from scratch, while reducing around 2x training cost. When transferring to several downstream classification datasets, TLEG surpasses existing initialization methods by a large margin (e.g., +6.87% on iNat 2019 and +7.66% on CIFAR-100). Under the situation where we need to produce models of varying depths adapting for different resource constraints, TLEG achieves comparable results while reducing around 19x parameters stored to initialize these models and around 5x pre-training costs, in contrast to the pre-training and fine-tuning approach. When transferring a fixed set of parameters to initialize different models, TLEG presents better flexibility and competitive performance while reducing around 2.9x parameters stored to initialize, compared to the pre-training approach

A Novel Fractional-Order PID Controller for Integrated Pressurized Water Reactor Based on Wavelet Kernel Neural Network Algorithm

This paper presents a novel wavelet kernel neural network (WKNN) with wavelet kernel function. It is applicable in online learning with adaptive parameters and is applied on parameters tuning of fractional-order PID (FOPID) controller, which could handle time delay problem of the complex control system. Combining the wavelet function and the kernel function, the wavelet kernel function is adopted and validated the availability for neural network. Compared to the conservative wavelet neural network, the most innovative character of the WKNN is its rapid convergence and high precision in parameters updating process. Furthermore, the integrated pressurized water reactor (IPWR) system is established by RELAP5, and a novel control strategy combining WKNN and fuzzy logic rule is proposed for shortening controlling time and utilizing the experiential knowledge sufficiently. Finally, experiment results verify that the control strategy and controller proposed have the practicability and reliability in actual complicated system

A new empirical approach to estimate temperature effects on strut loads in braced excavation

In deep excavation designs, strut loads play a key role to ensure excavation safety. During the construction, temperature fluctuation inevitably leads to a variation in strut loads. Therefore, how to quantitatively estimate the effects of temperature on strut loads is a matter of concern. In this study, the incremental changes in wall deflection due to temperature fluctuation were assumed to be piecewise linear. Based on the beam-on-elastic-foundation (BEF) model, an empirical approach accounting for the variation in temperature-induced strut loads at all levels was established. This model was further calibrated against a reported case study for a more precise predictive performance

Phylogenomics, taxonomy and morphological characters of the Microdochiaceae (Xylariales, Sordariomycetes)

Species of the family Microdochiaceae (Xylariales, Sordariomycetes) have been reported from worldwide, and collected from different plant hosts. The proposed new genus and two new species, viz., Macroidriella gen. nov., M. bambusae sp. nov. and Microdochium australe sp. nov., are based on multi-locus phylogenies from a combined dataset of ITS rDNA, LSU, RPB2 and TUB2 with morphological characteristics. Microdochium sinense has been collected from diseased leaves of Phragmites australis and this is the first report of the fungus on this host plant. Simultaneously, we annotated 10,372 to 11,863 genes, identified 4,909 single-copy orthologous genes, and conducted phylogenomic analysis based on genomic data. A gene family analysis was performed and it will expand the understanding of the evolutionary history and biodiversity of the Microdochiaceae. The detailed descriptions and illustrations of species are provided

Increased red cell distribution width in patients with slow coronary flow syndrome

OBJECTIVE: An elevated red cell distribution width has been recognized as a predictor of various cardiovascular diseases. Slow coronary flow syndrome is an important angiographic clinical entity with an unknown etiology. This study aimed to examine the relationship between red cell distribution width and the presence of slow coronary flow syndrome. METHODS: In total, 185 patients with slow coronary flow syndrome and 183 age- and gender-matched subjects with normal coronary flow (controls) were prospectively enrolled in this study. Red cell distribution width and C-reactive protein were measured upon admission, and the results were compared between the patients with slow coronary flow syndrome and normal controls. RESULTS: Red cell distribution width levels were significantly higher in the patients with slow coronary flow syndrome than the normal controls. Moreover, the data showed that the plasma C-reactive protein levels were also higher in the patients with slow coronary flow syndrome than in the normal controls. In addition, a multivariate analysis indicated that C-reactive protein and red cell distribution width were the independent variables most strongly associated with slow coronary flow syndrome. Finally, the red cell distribution width was positively correlated with C-reactive protein and mean thrombosis in the myocardial infarction frame counts of the patients with slow coronary flow syndrome. CONCLUSION: The data demonstrated that red cell distribution width levels are significantly higher and strongly positively correlated with both C-reactive protein and thrombosis in the myocardial infarction frame counts of patients with slow coronary flow syndrome. These findings suggest that red cell distribution width may be a useful marker for patients with slow coronary flow syndrome

MiR-9-1 Suppresses Cell Proliferation and Promotes Apoptosis by Targeting UHRF1 in Lung Cancer

Lung cancer is listed as the most common reason for cancer-related death all over the world despite diagnostic improvements and the development of chemotherapy and targeted therapies. MicroRNAs control both physiological and pathological processes including development and cancer. A microRNA-9 to 1 (miR-9 to 1) overexpression model in lung cancer cell lines was established and miR-9 to 1 was found to significantly suppress the proliferation rate in lung cancer cell lines, colony formation in vitro, and tumorigenicity in nude mice of A549 cells. Ubiquitin-like containing PHD and RING finger domains 1 (UHRF1) was then identified to direct target of miR-9 to 1. The inhibition of UHRF1 by miR-9 to 1 causes G1 arrest and p15, p16, and p21 were re-expressed in miR-9 to 1 group in mRNA level and protein level. Silence of UHRF1 expression in A549 cells resulted in the similar re-expression of p15, p16, p21 which is similar with miR-9 to 1 infection. Therefore, we concluded that UHRF1 is a new target for miR-9 to 1 to suppress cell proliferation by re-expression of tumor suppressors p15, p16, and p21 mediated by UHRF1

Memristors with diffusive dynamics as synaptic emulators for neuromorphic computing

The accumulation and extrusion of Ca2+ in the pre- and postsynaptic compartments play a critical role in initiating plastic changes in biological synapses. To emulate this fundamental process in electronic devices, we developed diffusive Ag-in-oxide memristors with a temporal response during and after stimulation similar to that of the synaptic Ca2+ dynamics. In situ high-resolution transmission electron microscopy and nanoparticle dynamics simulations both demonstrate that Ag atoms disperse under electrical bias and regroup spontaneously under zero bias because of interfacial energy minimization, closely resembling synaptic influx and extrusion of Ca2+, respectively. The diffusive memristor and its dynamics enable a direct emulation of both short- and long-term plasticity of biological synapses and represent a major advancement in hardware implementation of neuromorphic functionalities