IsoBN: Fine-Tuning BERT with Isotropic Batch Normalization

Fine-tuning pre-trained language models (PTLMs), such as BERT and its better variant RoBERTa, has been a common practice for advancing performance in natural language understanding (NLU) tasks. Recent advance in representation learning shows that isotropic (i.e., unit-variance and uncorrelated) embeddings can significantly improve performance on downstream tasks with faster convergence and better generalization. The isotropy of the pre-trained embeddings in PTLMs, however, is relatively under-explored. In this paper, we analyze the isotropy of the pre-trained [CLS] embeddings of PTLMs with straightforward visualization, and point out two major issues: high variance in their standard deviation, and high correlation between different dimensions. We also propose a new network regularization method, isotropic batch normalization (IsoBN) to address the issues, towards learning more isotropic representations in fine-tuning by dynamically penalizing dominating principal components. This simple yet effective fine-tuning method yields about 1.0 absolute increment on the average of seven NLU tasks.Comment: AAAI 202

Scavenging low-speed breeze wind energy using a triboelectric nanogenerator installed inside a square variable diameter channel

Over the recent years, triboelectric nanogenerator (TENG) have received widespread attention as a simple and efficient energy harvesting device. However, how to collect the breeze in daily life is an important issue that need to be solved for wind-powered triboelectric nanogenerator (W-TENG). Here, we propose a method of connecting a square variable diameter channel to the previously studied double-ended fixed W-TENG, which realizes the collection of energy in the breeze. As a result, after adding channels, the starting wind speed of W-TENG is optimized to as low as 0.4 m s−1, with an average output voltage of 6.1 V. This method not only enables W-TENG to start at the ultra-low wind speed, but also improves the output performance. When the external wind velocity is 2.0 m s−1, the output voltage is increased by 10.6 times after adding the channel structure. This work provides a good strategy for collecting the breeze without changing the original structure of the W-TENG, fully demonstrating the advantages of energy harvesting under the low wind velocity.</p

Scavenging low-speed breeze wind energy using a triboelectric nanogenerator installed inside a square variable diameter channel

Over the recent years, triboelectric nanogenerator (TENG) have received widespread attention as a simple and efficient energy harvesting device. However, how to collect the breeze in daily life is an important issue that need to be solved for wind-powered triboelectric nanogenerator (W-TENG). Here, we propose a method of connecting a square variable diameter channel to the previously studied double-ended fixed W-TENG, which realizes the collection of energy in the breeze. As a result, after adding channels, the starting wind speed of W-TENG is optimized to as low as 0.4 m s−1, with an average output voltage of 6.1 V. This method not only enables W-TENG to start at the ultra-low wind speed, but also improves the output performance. When the external wind velocity is 2.0 m s−1, the output voltage is increased by 10.6 times after adding the channel structure. This work provides a good strategy for collecting the breeze without changing the original structure of the W-TENG, fully demonstrating the advantages of energy harvesting under the low wind velocity.</p

Predicting Continuous Locomotion Modes via Multidimensional Feature Learning from sEMG

Walking-assistive devices require adaptive control methods to ensure smooth transitions between various modes of locomotion. For this purpose, detecting human locomotion modes (e.g., level walking or stair ascent) in advance is crucial for improving the intelligence and transparency of such robotic systems. This study proposes Deep-STF, a unified end-to-end deep learning model designed for integrated feature extraction in spatial, temporal, and frequency dimensions from surface electromyography (sEMG) signals. Our model enables accurate and robust continuous prediction of nine locomotion modes and 15 transitions at varying prediction time intervals, ranging from 100 to 500 ms. In addition, we introduced the concept of 'stable prediction time' as a distinct metric to quantify prediction efficiency. This term refers to the duration during which consistent and accurate predictions of mode transitions are made, measured from the time of the fifth correct prediction to the occurrence of the critical event leading to the task transition. This distinction between stable prediction time and prediction time is vital as it underscores our focus on the precision and reliability of mode transition predictions. Experimental results showcased Deep-STP's cutting-edge prediction performance across diverse locomotion modes and transitions, relying solely on sEMG data. When forecasting 100 ms ahead, Deep-STF surpassed CNN and other machine learning techniques, achieving an outstanding average prediction accuracy of 96.48%. Even with an extended 500 ms prediction horizon, accuracy only marginally decreased to 93.00%. The averaged stable prediction times for detecting next upcoming transitions spanned from 28.15 to 372.21 ms across the 100-500 ms time advances.Comment: 10 pages,7 figure

Distribution of six phenolic acids and soil nutrient relationships during litter decomposition in Rhododendron forests

Litter decomposition is an essential process in nutrient cycling in terrestrial ecosystems. Phenolic acids have an allelopathic effect on the natural regeneration of rhododendron forests, which was recently observed in rhododendron forests in Southwest China. We investigated the distribution of phenolic acids and their relationship with soil nutrients during litter decomposition to provide a reference for the subsequent artificial management of rhododendron forests. High-performance liquid chromatography (HPLC) was used to analyze the contents of six phenolic acids in the litter layer, humus layer, and soil layer during litter decomposition. During the first 3 three months, the rapid decomposition of the litter from two early flowering rhododendron species released large amounts of phenolic acids, and the lower litter layer decomposed rapidly. In addition, the total phenolic acid content in the litter decreased by 257.60 μg/g (Rhododendron siderophyllum) and 53.12 μg/g (Rhododendron annae) in the first three 3 months. During subsequent litter decomposition, the protocatechuic acid content in the litters of Rhododendron siderophyllum ranged from 20.15 μg/g to 39.12 μg/g, and the ferulic acid content was 10.70 μg/g to –33.79 μg/g. The protocatechuic acid and ferulic acid contents in the litter of Rhododendron annae were in the ranges of 10.88—19.68 μg/g and 10.75—18.00 μg/g, respectively. The contents of these two phenolic acids and the trend of decomposition were different from those of gallic acid, chlorogenic acid, caffeic acid and syringic acid. The distribution of phenolic acids was influenced by soil organic matter (SOM), soil ammonium nitrogen (NH4+), soil nitrate nitrogen (NO3–) and soil available phosphorus (AP). The results indicate seasonal variations in phenolic acid release during litter decomposition. The amount of phenolic acid in the litter decreased after 18 months of decomposition, but it returned to the previous level in the soil and the humus after different trends. More research into the metabolism of phenolic acids is needed