AWTE-BERT:Attending to Wordpiece Tokenization Explicitly on BERT for Joint Intent Classification and SlotFilling

Intent classification and slot filling are two core tasks in natural language understanding (NLU). The interaction nature of the two tasks makes the joint models often outperform the single designs. One of the promising solutions, called BERT (Bidirectional Encoder Representations from Transformers), achieves the joint optimization of the two tasks. BERT adopts the wordpiece to tokenize each input token into multiple sub-tokens, which causes a mismatch between the tokens and the labels lengths. Previous methods utilize the hidden states corresponding to the first sub-token as input to the classifier, which limits performance improvement since some hidden semantic informations is discarded in the fine-tune process. To address this issue, we propose a novel joint model based on BERT, which explicitly models the multiple sub-tokens features after wordpiece tokenization, thereby generating the context features that contribute to slot filling. Specifically, we encode the hidden states corresponding to multiple sub-tokens into a context vector via the attention mechanism. Then, we feed each context vector into the slot filling encoder, which preserves the integrity of the sentence. Experimental results demonstrate that our proposed model achieves significant improvement on intent classification accuracy, slot filling F1, and sentence-level semantic frame accuracy on two public benchmark datasets. The F1 score of the slot filling in particular has been improved from 96.1 to 98.2 (2.1% absolute) on the ATIS dataset

Taming Gradient Variance in Federated Learning with Networked Control Variates

Federated learning, a decentralized approach to machine learning, faces significant challenges such as extensive communication overheads, slow convergence, and unstable improvements. These challenges primarily stem from the gradient variance due to heterogeneous client data distributions. To address this, we introduce a novel Networked Control Variates (FedNCV) framework for Federated Learning. We adopt the REINFORCE Leave-One-Out (RLOO) as a fundamental control variate unit in the FedNCV framework, implemented at both client and server levels. At the client level, the RLOO control variate is employed to optimize local gradient updates, mitigating the variance introduced by data samples. Once relayed to the server, the RLOO-based estimator further provides an unbiased and low-variance aggregated gradient, leading to robust global updates. This dual-side application is formalized as a linear combination of composite control variates. We provide a mathematical expression capturing this integration of double control variates within FedNCV and present three theoretical results with corresponding proofs. This unique dual structure equips FedNCV to address data heterogeneity and scalability issues, thus potentially paving the way for large-scale applications. Moreover, we tested FedNCV on six diverse datasets under a Dirichlet distribution with {\alpha} = 0.1, and benchmarked its performance against six SOTA methods, demonstrating its superiority.Comment: 14 page

Graph Learning and Its Applications: A Holistic Survey

Graph learning is a prevalent domain that endeavors to learn the intricate relationships among nodes and the topological structure of graphs. These relationships endow graphs with uniqueness compared to conventional tabular data, as nodes rely on non-Euclidean space and encompass rich information to exploit. Over the years, graph learning has transcended from graph theory to graph data mining. With the advent of representation learning, it has attained remarkable performance in diverse scenarios, including text, image, chemistry, and biology. Owing to its extensive application prospects, graph learning attracts copious attention from the academic community. Despite numerous works proposed to tackle different problems in graph learning, there is a demand to survey previous valuable works. While some researchers have perceived this phenomenon and accomplished impressive surveys on graph learning, they failed to connect related objectives, methods, and applications in a more coherent way. As a result, they did not encompass current ample scenarios and challenging problems due to the rapid expansion of graph learning. Different from previous surveys on graph learning, we provide a holistic review that analyzes current works from the perspective of graph structure, and discusses the latest applications, trends, and challenges in graph learning. Specifically, we commence by proposing a taxonomy from the perspective of the composition of graph data and then summarize the methods employed in graph learning. We then provide a detailed elucidation of mainstream applications. Finally, based on the current trend of techniques, we propose future directions.Comment: 20 pages, 7 figures, 3 table

Mortality predicting models for patients with infective endocarditis: a machine learning approach

Background Infective endocarditis (IE) is a fatal cardiovascular disease with varied clinical manifestations but rapid progression. A series of existing risk models helped identify IE patients with high risk, but the imperfect predictive performance and limited application called for better predictive systems. Methods The single-centered, retrospective observational study applied four machine learning methods for predictive model construction: LASSO logistic regression, random forest (RF), support vector machine (SVM), and k-nearest neighbors (KNN). A 10-fold cross-validated area under the receiver operating characteristic curve (AUC-ROC) was used for performance evaluation. Results A total of 1705 patients with IE were enrolled in the study, with 119 in-hospital deaths and 178 deaths after 6-month follow-up. RF achieved the highest AUC-ROCs for in-hospital and six-month mortality prediction (in-hospital: 0.83, 6-month: 0.85). RF was also applied to assess variable importance. The following variables were selected by RF as top important predictors for both in-hospital and six-month mortality prediction: total bilirubin, N-terminal pro-B-type natriuretic peptide, albumin, diastolic blood pressure, fasting blood glucose, uric acid, and age. Conclusions A risk model with machine learning approach was integrated in purpose of prognosis prediction in IE patients, helping rapid risk stratification and in-time management clinically. Clinical trial number Not applicable

Theoretical analysis of a regenerative supercritical carbon dioxide Brayton cycle/organic Rankine cycle dual loop for waste heat recovery of a diesel/natural gas dual-fuel engine

Supercritical carbon dioxide Brayton cycle is considered one of the most promising systems for waste heat recovery of engines because of its compactness and high energy efficiency. To further improve the fuel utilization ratio and solve the difficulties of waste heat recovery of high temperature exhaust gas, a regenerative supercritical carbon dioxide Brayton cycle/organic Rankine cycle dual loop is proposed for cascade utilization of exhaust heat from a dual-fuel engine. The regenerative supercritical carbon dioxide Brayton cycle of the proposed system is powered by the waste heat contained in the exhaust gas. The working fluid in the organic Rankine cycle is pre-heated by CO2 exiting the regenerator and then further heated by the residual heat of the exhaust gas. The flow rates of the working fluids in both sub cycles are adjusted to match the waste heat recovery system to respond to the changing conditions of the dual-fuel engine. The results revealed that the maximum net power output of this system is up to 40.88 kW, thus improving the dual-fuel engine power output by 6.78%. Therefore, such a regenerative supercritical carbon dioxide Brayton cycle/organic Rankine cycle dual loop system design enables the thorough recovery of high temperature exhaust heat, leading to higher energy efficiency and lower fuel consumption of the engine

Highly-stable, flexible delivery of microjoule-level ultrafast pulses in vacuumized anti-resonant hollow-core fibers for active synchronization

We demonstrate the stable and flexible light delivery of multi-{\mu}J, sub-200-fs pulses over a ~10-m-long vacuumized anti-resonant hollow-core fiber (AR-HCF), which was successfully used for high-performance pulse synchronization. Compared with the pulse train launched into the AR-HCF, the transmitted pulse train out of the fiber exhibits excellent stabilities in pulse power and spectrum, with pointing stability largely improved. The walk-off between the fiber-delivery and the other free-space-propagation pulse trains, in an open loop, was measured to be <6 fs root-mean-square (RMS) over 90 minutes, corresponding to a relative optical-path variation of <2x10-7. This walk-off can be further suppressed to ~2 fs RMS simply using an active control loop, highlighting the great application potentials of this AR-HCF set-up in large-scale laser and accelerator facilities

Effect of Chitosan Coating with Different Molecular Weights on the Storage Quality of Postharvest Passion Fruit (Passiflora edulis Sims)

To study the preservation effect of chitosan coating with different molecular weights on postharvest passion fruit, the "Qinmi No.9" was coated with chitosan of molecular weights of 30, 50, 100, 150 and 200 kDa (1.5%, w/v) to determine the quality of passion fruit during storage. The results showed that chitosan coating with different molecular weights was able to delay the shrinkage and yellowing, reduce the weight loss rate and inhibit the decay of passion fruit. Moreover, chitosan with a larger molecular weight was more conducive to delaying the ripening and senescence of passion fruit, as well as reducing shrinkage, and decay. At the end of storage, the weight loss of fruits coated with 200 kDa chitosan was nearly 10% less than that coated with 30 kDa chitosan, and the fruits coated with 150 and 200 kDa chitosan did not decay. The lower molecular weight (30 and 50 kDa) and higher molecular weight (150 kDa) chitosan were more effective in inhibiting weight loss, total soluble solids and soluble sugar metabolism, and maintaining titratable acid, flavonoid and total phenol contents of fruit during storage. The chitosan with 150 kDa had the best effect in maintaining the vitamin C content, which was 1.12 times higher than the control group at the end of storage. In conclusion, chitosan with different molecular weights was effective to delay senescence, slow down water loss and shrink of passion fruit and maintain the quality, chitosan with 150 kDa was more suitable to maintain the quality of postharvest passion fruit

Genetic diversity fuels gene discovery for tobacco and alcohol use

Tobacco and alcohol use are heritable behaviours associated with 15% and 5.3% of worldwide deaths, respectively, due largely to broad increased risk for disease and injury(1-4). These substances are used across the globe, yet genome-wide association studies have focused largely on individuals of European ancestries(5). Here we leveraged global genetic diversity across 3.4 million individuals from four major clines of global ancestry (approximately 21% non-European) to power the discovery and fine-mapping of genomic loci associated with tobacco and alcohol use, to inform function of these loci via ancestry-aware transcriptome-wide association studies, and to evaluate the genetic architecture and predictive power of polygenic risk within and across populations. We found that increases in sample size and genetic diversity improved locus identification and fine-mapping resolution, and that a large majority of the 3,823 associated variants (from 2,143 loci) showed consistent effect sizes across ancestry dimensions. However, polygenic risk scores developed in one ancestry performed poorly in others, highlighting the continued need to increase sample sizes of diverse ancestries to realize any potential benefit of polygenic prediction.Peer reviewe