FF2: A Feature Fusion Two-Stream Framework for Punctuation Restoration

To accomplish punctuation restoration, most existing methods focus on introducing extra information (e.g., part-of-speech) or addressing the class imbalance problem. Recently, large-scale transformer-based pre-trained language models (PLMS) have been utilized widely and obtained remarkable success. However, the PLMS are trained on the large dataset with marks, which may not fit well with the small dataset without marks, causing the convergence to be not ideal. In this study, we propose a Feature Fusion two-stream framework (FF2) to bridge the gap. Specifically, one stream leverages a pre-trained language model to capture the semantic feature, while another auxiliary module captures the feature at hand. We also modify the computation of multi-head attention to encourage communication among heads. Then, two features with different perspectives are aggregated to fuse information and enhance context awareness. Without additional data, the experimental results on the popular benchmark IWSLT demonstrate that FF2 achieves new SOTA performance, which verifies that our approach is effective.Comment: 5pages. arXiv admin note: substantial text overlap with arXiv:2203.1248

TLM: Token-Level Masking for Transformers

Structured dropout approaches, such as attention dropout and DropHead, have been investigated to regularize the multi-head attention mechanism in Transformers. In this paper, we propose a new regularization scheme based on token-level rather than structure-level to reduce overfitting. Specifically, we devise a novel Token-Level Masking (TLM) training strategy for Transformers to regularize the connections of self-attention, which consists of two masking techniques that are effective and easy to implement. The underlying idea is to manipulate the connections between tokens in the multi-head attention via masking, where the networks are forced to exploit partial neighbors' information to produce a meaningful representation. The generality and effectiveness of TLM are thoroughly evaluated via extensive experiments on 4 diversified NLP tasks across 18 datasets, including natural language understanding benchmark GLUE, ChineseGLUE, Chinese Grammatical Error Correction, and data-to-text generation. The results indicate that TLM can consistently outperform attention dropout and DropHead, e.g., it increases by 0.5 points relative to DropHead with BERT-large on GLUE. Moreover, TLM can establish a new record on the data-to-text benchmark Rotowire (18.93 BLEU). Our code will be publicly available at https://github.com/Young1993/tlm.Comment: 13 pages. Accepted by EMNLP2023 main conferenc

A stable sandwich' system of Surface-Enhanced Resonance Raman Scattering for the analysis of -carotenes in a photosynthetic pigment-protein complex

In plants, Photosystem I (PSI) is composed of a core complex and a membrane-associated antenna complex light-harvesting complex I that captures light and funnels its energy to the core complex. To obtain Raman structural information on -carotenes embedded in the PSI core complex, a sandwich' system of roughened silver slice: target protein complexes: single silver nanoparticles was fabricated for Surface-Enhanced Resonance Raman Scattering (SERRS) measurements. This study provided a method to overcome spectral irreproducibility, which is the main drawback of Surface-Enhanced Raman Scattering/SERRS-based studies. The Raman spectra of -carotenes embedded in the PSI core complex can be obtained at very low sample concentrations (1-5 mu g Chl/ml) and high signal/noise ratios. The -carotenes in the spinach PSI core complex were predominantly all-trans configuration. The membrane protein-mediated adsorption of silver nanoparticles induced the uniform distribution of a large number of single nanoparticles, which contributed to achieving highly reproducible SERRS spectra. This study is the first to apply single silver nanoparticle-based SERRS analysis in membrane proteins. Copyright (c) 2013 John Wiley & Sons, Ltd

The Brain Activity in Brodmann Area 17: A Potential Bio-Marker to Predict Patient Responses to Antiepileptic Drugs

<div><p>In this study, we aimed to predict newly diagnosed patient responses to antiepileptic drugs (AEDs) using resting-state functional magnetic resonance imaging tools to explore changes in spontaneous brain activity. We recruited 21 newly diagnosed epileptic patients, 8 drug-resistant (DR) patients, 11 well-healed (WH) patients, and 13 healthy controls. After a 12-month follow-up, 11 newly diagnosed epileptic patients who showed a poor response to AEDs were placed into the seizures uncontrolled (SUC) group, while 10 patients were enrolled in the seizure-controlled (SC) group. By calculating the amplitude of fractional low-frequency fluctuations (fALFF) of blood oxygen level-dependent signals to measure brain activity during rest, we found that the SUC patients showed increased activity in the bilateral occipital lobe, particularly in the cuneus and lingual gyrus compared with the SC group and healthy controls. Interestingly, DR patients also showed increased activity in the identical cuneus and lingual gyrus regions, which comprise Brodmann’s area 17 (BA17), compared with the SUC patients; however, these abnormalities were not observed in SC and WH patients. The receiver operating characteristic (ROC) curves indicated that the fALFF value of BA17 could differentiate SUC patients from SC patients and healthy controls with sufficient sensitivity and specificity prior to the administration of medication. Functional connectivity analysis was subsequently performed to evaluate the difference in connectivity between BA17 and other brain regions in the SUC, SC and control groups. Regions nearby the cuneus and lingual gyrus were found positive connectivity increased changes or positive connectivity changes with BA17 in the SUC patients, while remarkably negative connectivity increased changes or positive connectivity decreased changes were found in the SC patients. Additionally, default mode network (DMN) regions showed negative connectivity increased changes or negative changes with BA17 in the SUC patients. The abnormal increased in BA17 activity may be a key point that plays a substantial role in facilitating seizure onset.</p></div

Map of fALFF differences among the SUC, SC, DR, WH and control groups.

<p>There were significant fALFF differences among the five groups in the bilateral cuneus, lingual gyrus, inferior/middle occipital gyrus, calcarine, middle temporal-occipital area, fusiform, subcortical structure of left occipital lobe, subcortical structure of right temporal lobe, right posterior cingulated, and right cerebellum posterior lobe. The statistical threshold was set at P < 0.05 and a cluster size > 4158 mm³, which corresponded to a corrected P < 0.05.</p

Spatial overlapping maps and scatter plots showing fALFF values of overlaps.

<p><b>A:</b> SUC vs. CON and DR vs. CON. Overlap regions account for 20% of SUC <i>vs</i>. controls, 41.7% of DR <i>vs</i>. controls, and 15.9% of total respectively. The yellow parts represents brain regions with common fALFF changes between SUC <i>vs</i>. controls and DR <i>vs</i>. controls including the bilateral lingual gyrus, cuneus, inferior occipital gyrus, middle occipital gyrus, right superior occipital gyrus, subcortical structure of left occipital lobe, subcortical structure of right temporal lobe, and left fusiform. The red parts show regions with fALFF differences from comparison of SUC vs. CON only. The blue parts show regions with fALFF differences from comparison of DR vs. CON only. <b>B:</b> SUC vs. SC and DR vs. WH. Overlap regions account for 27.2% of SUC <i>vs</i>. SC, 24.7% of DR <i>vs</i>. WH, and 14.9% of total respectively. The yellow parts represents brain regions with common fALFF changes between SUC <i>vs</i>. SC and DR <i>vs</i>. WH were found in the bilateral middle occipital gyrus, fusiform, lingual gyrus, the right superior occipital gyrus, cuneus and cerebellum posterior lobe. The red parts show regions with fALFF differences from comparison of SUC vs. SC only. The blue parts show regions with fALFF differences from comparison of DR vs. WH only. All comparisons were restrained in the ANOVA mask. <b>C:</b> fALFF values of overlap with common brain activity changes between SUC vs. CON and DR vs. CON. The error bar represents the standard deviation. * P< 0.001. <b>D:</b> fALFF values of overlap with common brain activity changes between SUC vs. SC and DR vs. WH. The error bar represents the standard deviation. * P = 0.001, * *P< 0.001.</p

Maps of fALFF differences.

<p><b>A:</b> SUC vs. control. Compared with healthy controls, the SUC patients showed significantly increased fALFF values in the warm color regions, including the bilateral cuneus, bilateral lingual gyrus, bilateral superior/middle/inferior occipital gyrus, and right posterior cingulate. <b>B:</b> SUC vs. SC. Compared with SC patients, the SUC patients showed significantly increased fALFF values in the warm color regions of the bilateral cuneus, bilateral lingual gyrus, bilateral middle temporal-occipital area, and right fusiform gyrus. <b>C:</b> DR vs. controls. The DR patients showed significantly increased fALFF values in the warm color regions of the bilateral cuneus, bilateral middle occipital gyrus, bilateral fusiform, and right middle temporal-occipital area. <b>D:</b> DR vs. WH. Compared with WH patients, the DR patients showed significantly increased fALFF values in the warm color regions of the left cuneus, bilateral fusiform, and right middle occipital gyrus. <b>E:</b> SC vs. CON. Compared with the healthy controls, the SC patients showed significantly increased fALFF values in the warm color region of the left inferior occipital gyrus. In contrast, the cold color regions in the right fusiform gyrus represent the area with decreased fALFF values in SC patients compared with controls. <b>F:</b> WH vs. CON. The WH patients showed only showed decreased fALFF values in the cold color region of the right fusiform gyrus. The statistical threshold was set at P < 0.05 with a cluster size > 351 mm³, which corresponded to a corrected P < 0.05.</p