Unlocking Emergent Modularity in Large Language Models

Modular Neural Networks (MNNs) demonstrate various advantages over monolithic models. Existing MNNs are generally $\textit{explicit}$: their modular architectures are pre-defined, with individual modules expected to implement distinct functions. Recent works reveal that there exists $\textit{implicit}$ modularity in standard pre-trained transformers, namely $\textit{Emergent Modularity}$. They indicate that such modular structures spontaneously exhibit during the early pre-training phase. Despite the benefits of modularity, most Language Models (LMs) are still treated as monolithic models in the pre-train and fine-tune paradigm, with their emergent modularity locked and underutilized. In this work, focusing on unlocking the emergent modularity in LMs, we showcase that standard LMs could be fine-tuned as their Mixture-of-Expert (MoEs) counterparts without introducing any extra parameters. Such MoEs are derived from emergent modularity and are referred to as Emergent MoEs (EMoE). Our experiments demonstrate that fine-tuning EMoE effectively improves downstream in-domain and out-of-domain generalization compared with vanilla fine-tuning. Our analysis and ablation studies further illustrate that it is robust to various configurations and can scale up to Large Language Models (i.e., Llama2-7B and Llama-30B). Code is available at https://github.com/qiuzh20/EMoE.Comment: NAACL2024 Main Conferenc

Maps of cropping patterns in China during 2015–2021

Multiple cropping is a widespread approach for intensifying crop production through rotations of diverse crops. Maps of cropping intensity with crop descriptions are important for supporting sustainable agricultural management. As the most populated country, China ranked first in global cereal production and the percentages of multiple-cropped land are twice of the global average. However, there are no reliable updated national-scale maps of cropping patterns in China. Here we present the first recent annual 500-m MODIS-based national maps of multiple cropping systems in China using phenologybased mapping algorithms with pixel purity-based thresholds, which provide information on cropping intensity with descriptions of three staple crops (maize, paddy rice, and wheat). The produced cropping patterns maps achieved an overall accuracy of 89% based on ground truth data, and a good agreement with the statistical data (R2 ≥ 0.89). The China Cropping Pattern maps (ChinaCP) are available for public download online. Cropping patterns maps in China and other countries with finer resolutions can be produced based on Sentinel-2 Multispectral Instrument (MSI) images using the shared code

CLAP: Learning Transferable Binary Code Representations with Natural Language Supervision

Binary code representation learning has shown significant performance in binary analysis tasks. But existing solutions often have poor transferability, particularly in few-shot and zero-shot scenarios where few or no training samples are available for the tasks. To address this problem, we present CLAP (Contrastive Language-Assembly Pre-training), which employs natural language supervision to learn better representations of binary code (i.e., assembly code) and get better transferability. At the core, our approach boosts superior transfer learning capabilities by effectively aligning binary code with their semantics explanations (in natural language), resulting a model able to generate better embeddings for binary code. To enable this alignment training, we then propose an efficient dataset engine that could automatically generate a large and diverse dataset comprising of binary code and corresponding natural language explanations. We have generated 195 million pairs of binary code and explanations and trained a prototype of CLAP. The evaluations of CLAP across various downstream tasks in binary analysis all demonstrate exceptional performance. Notably, without any task-specific training, CLAP is often competitive with a fully supervised baseline, showing excellent transferability. We release our pre-trained model and code at https://github.com/Hustcw/CLAP

Seasonality of the transmissibility of hand, foot and mouth disease: a modelling study in Xiamen City, China.

This study attempts to figure out the seasonality of the transmissibility of hand, foot and mouth disease (HFMD). A mathematical model was established to calculate the transmissibility based on the reported data for HFMD in Xiamen City, China from 2014 to 2018. The transmissibility was measured by effective reproduction number (Reff) in order to evaluate the seasonal characteristics of HFMD. A total of 43 659 HFMD cases were reported in Xiamen, for the period 2014 to 2018. The median of annual incidence was 221.87 per 100 000 persons (range: 167.98/100,000-283.34/100 000). The reported data had a great fitting effect with the model (R2 = 0.9212, P < 0.0001), it has been shown that there are two epidemic peaks of HFMD in Xiamen every year. Both incidence and effective reproduction number had seasonal characteristics. The peak of incidence, 1-2 months later than the effective reproduction number, occurred in Summer and Autumn, that is, June and October each year. Both the incidence and transmissibility of HFMD have obvious seasonal characteristics, and two annual epidemic peaks as well. The peak of incidence is 1-2 months later than Reff

Prediction of Screw Loosening After Dynamic Pedicle Screw Fixation With Lumbar Polyetheretherketone Rods Using Magnetic Resonance Imaging-Based Vertebral Bone Quality Score

Objective To investigate the correlation between magnetic resonance imaging-based vertebral bone quality (VBQ) score and screw loosening after dynamic pedicle screw fixation with polyetheretherketone (PEEK) rods, and evaluate its predictive value. Methods A retrospective analysis was conducted on the patients who underwent dynamic pedicle screw fixation with PEEK rods from March 2017 to June 2022. Data on age, sex, body mass index, hypertension, diabetes, hyperlipidemia history, long-term smoking, alcohol consumption, VBQ score, L1–4 average Hounsfield unit (HU) value, surgical fixation length, and the lowest instrumented vertebra were collected. Logistic regression analysis was employed to assess the relationship between VBQ score and pedicle screw loosening (PSL). Results A total of 24 patients experienced PSL after surgery (20.5%). PSL group and non-PSL group showed statistical differences in age, number of fixed segments, fixation to the sacrum, L1–4 average HU value, and VBQ score (p < 0.05). The VBQ score in the PSL group was higher than that in the non-PSL group (3.56 ± 0.45 vs. 2.77 ± 0.31, p < 0.001). In logistic regression analysis, VBQ score (odds ratio, 3.425; 95% confidence interval, 1.552–8.279) were identified as independent risk factors for screw loosening. The area under the receiver operating characteristic curve for VBQ score predicting PSL was 0.819 (p < 0.05), with the optimal threshold of 3.15 (sensitivity, 83.1%; specificity, 80.5%). Conclusion The VBQ score can independently predict postoperative screw loosening in patients undergoing lumbar dynamic pedicle screw fixation with PEEK rods, and its predictive value is comparable to HU value

Evolutionary origin of genomic structural variations in domestic yaks

Yak has been subject to natural selection, human domestication and interspecific introgression during its evolution. However, genetic variants favored by each of these processes have not been distinguished previously. We constructed a graph-genome for 47 genomes of 7 cross-fertile bovine species. This allowed detection of 57,432 high-resolution structural variants (SVs) within and across the species, which were genotyped in 386 individuals. We distinguished the evolutionary origins of diverse SVs in domestic yaks by phylogenetic analyses. We further identified 334 genes overlapping with SVs in domestic yaks that bore potential signals of selection from wild yaks, plus an additional 686 genes introgressed from cattle. Nearly 90% of the domestic yaks were introgressed by cattle. Introgression of an SV spanning the KIT gene triggered the breeding of white domestic yaks. We validated a significant association of the selected stratified SVs with gene expression, which contributes to phenotypic variations. Our results highlight that SVs of different origins contribute to the phenotypic diversity of domestic yaks

Blind separation of PCMA signals based on neural network

For paired carrier multiple access (PCMA) signals, a new single-channel blind separation on neural network was proposed. Firstly, the sample waveforms (three symbols) which contains different bit information are constructed, secondly, the time-frequency spectrum of each sample under the different influences of the trailing symbols is Intercepted, finally, the characteristic data of the spectrum as the input data, and the two-bit sequence in each sample as the output data to be trained, network trains these data repeatedly to complete the construction of separation model. The receiver carries on window truncation to the time-frequency spectrum of PCMA signal, neural network recognize the characteristic data of these spectrums to realizes the separation of bit sequences. Experimental results show that this algorithm has lower complexity than PSP algorithm, and the accuracy of it is close to PSP algorithm (L=5)