Improving the Robustness of Transformer-based Large Language Models with Dynamic Attention

Transformer-based models, such as BERT and GPT, have been widely adopted in natural language processing (NLP) due to their exceptional performance. However, recent studies show their vulnerability to textual adversarial attacks where the model's output can be misled by intentionally manipulating the text inputs. Despite various methods that have been proposed to enhance the model's robustness and mitigate this vulnerability, many require heavy consumption resources (e.g., adversarial training) or only provide limited protection (e.g., defensive dropout). In this paper, we propose a novel method called dynamic attention, tailored for the transformer architecture, to enhance the inherent robustness of the model itself against various adversarial attacks. Our method requires no downstream task knowledge and does not incur additional costs. The proposed dynamic attention consists of two modules: (I) attention rectification, which masks or weakens the attention value of the chosen tokens, and (ii) dynamic modeling, which dynamically builds the set of candidate tokens. Extensive experiments demonstrate that dynamic attention significantly mitigates the impact of adversarial attacks, improving up to 33\% better performance than previous methods against widely-used adversarial attacks. The model-level design of dynamic attention enables it to be easily combined with other defense methods (e.g., adversarial training) to further enhance the model's robustness. Furthermore, we demonstrate that dynamic attention preserves the state-of-the-art robustness space of the original model compared to other dynamic modeling methods

GaLileo: General Linear Relaxation Framework for Tightening Robustness Certification of Transformers

Transformers based on attention mechanisms exhibit vulnerability to adversarial examples, posing a substantial threat to the security of their applications. Aiming to solve this problem, the concept of robustness certification is introduced to formally ascertain the presence of any adversarial example within a specified region surrounding a given sample. However, prior works have neglected the dependencies among inputs of softmax (the most complex function in attention mechanisms) during linear relaxations. This oversight has consequently led to imprecise certification results. In this work, we introduce GaLileo, a general linear relaxation framework designed to certify the robustness of Transformers. GaLileo effectively surmounts the trade-off between precision and efficiency in robustness certification through our innovative n-dimensional relaxation approach. Notably, our relaxation technique represents a pioneering effort as the first linear relaxation for n-dimensional functions such as softmax. Our novel approach successfully transcends the challenges posed by the curse of dimensionality inherent in linear relaxations, thereby enhancing linear bounds by incorporating input dependencies. Our evaluations encompassed a thorough analysis utilizing the SST and Yelp datasets along with diverse Transformers of different depths and widths. The experimental results demonstrate that, as compared to the baseline method CROWN-BaF, GaLileo achieves up to 3.24 times larger certified radii while requiring similar running times. Additionally, GaLileo successfully attains certification for Transformers' robustness against multi-word lp perturbations, marking a notable accomplishment in this field

Research on feature extraction and classification of AE signals of fibers' tensile failure based on HHT and SVM

In order to study the feature extraction and recognition method of fibers' tensile failure, AE technology is used to collect AE signals of fiber bundle's tensile fracture of two kinds of fibers of Aramid 1313 and viscose. A transform called wavelet is used to deal with the signals to reduce noise. A method called Hilbert-Huang transform (HHT) is used to extract characteristic frequencies of the signals after the noise is reduced. And a classification method called Least Squares support vector machines (LSSVM) is used for the classification and recognition of characteristic frequencies of the two kinds of fibers. The results show that wavelet de-noise method can reduce some noise of the signals. Hilbert spectrum can reflect fracture circumstances of the two kinds of fibers in the time dimension to some extent. Characteristic frequencies' extraction can be done from marginal spectrum. The LSSVM can be used for the classification and recognition of characteristic frequencies. The recognition rates of Aramid 1313 and viscose reach 40%, 80% respectively, and the total recognition rate reaches 60%

Optical field simulation of edge coupled terahertz quantum well photodetectors

By finite difference time domain (FDTD) simulation, we report on the optical field distribution within edge coupled terahertz quantum well photodetectors (THz QWPs) in detail. The coupling efficiency of THz QWP structures are studied from three aspects, including the electrode geometry, the position of the active region and the coupling angle. According to the simulation results, proper electrode geometry is suggested in different frequency region, and the optimal position of active region and coupling angle are presented. These results provide a useful guidance for the design and fabrication of the edge coupled THz QWP

Monkfish (Lophius litulon) Peptides Ameliorate High-Fat-Diet-Induced Nephrotoxicity by Reducing Oxidative Stress and Inflammation via Regulation of Intestinal Flora

Background: Renal damage and intestinal flora imbalance due to lipotoxicity are particularly significant in terms of oxidative stress and inflammation, which can be alleviated with bioactive peptides. The monkfish (Lophius litulon) is rich in proteins, which can be used as a source of quality bioactive peptides. This study aimed to examine the protective effect of monkfish peptides on renal injury and their potential role in regulating gut microbiota. Methods: Monkfish meat was hydrolyzed using neutral protease and filtered, and the component with the highest elimination rate of 2,2-diphenyl-1-picrylhydrazyl was named lophius litulon peptides (LPs). Lipid nephrotoxicity was induced via high-fat diet (HFD) feeding for 8 weeks and then treated with LPs. Oxidative stress, inflammatory factors, and intestinal flora were evaluated. Results: LP (200 mg/kg) therapy reduced serum creatinine, uric acid, and blood urea nitrogen levels by 49.5%, 31.6%, and 31.6%, respectively. Renal vesicles and tubules were considerably improved with this treatment. Moreover, the activities of superoxide dismutase, glutathione peroxidase, and total antioxidant capacity increased significantly by 198.7%, 167.9%, 61.5%, and 89.4%, respectively. LPs attenuated the upregulation of HFD-induced Toll-like receptor 4 and phospho-nuclear factor-kappa B and increased the protein levels of heme oxygenase 1, nicotinamide quinone oxidoreductase 1, and nuclear factor erythroid 2-related factor 2. The dysbiosis of intestinal microbiota improved after LP treatment. Conclusions: LPs significantly improve antioxidant activity, reduce inflammatory cytokine levels, and regulate intestinal dysbiosis. Thus, LPs are potential compounds that can alleviate HFD-induced renal lipotoxicity

Growth trajectory of full-term small-for-gestational-age infants: a 3-year longitudinal study in China

Objective Small-for-gestational-age (SGA) infants are at risk of impaired growth and developmental outcomes, even for those who were born at full term. The growth trajectory of full-term SGA infants remains unknown. Therefore, this study aimed to evaluate the growth trajectory of full-term SGA infants from birth to 3 years old in East China.Methods Full-term SGA infants were followed up from birth to 3 years old. The weight and length were measured at 3, 6, 12, 18, 24, 30 and 36 months. Rate of catch-up growth and rates of growth deviations including short stature, emaciation, underweight, overweight and obesity, were calculated at different time points. Latent class analysis was applied to describe growth trajectories from birth to 36 months.Results A total of 816 full-term SGA infants were enrolled in this study and 303 had complete follow-up data at 3, 6, 12, 18, 24, 30 and 36 months. At 24 months, the rate of catch-up growth was 42.4% in girls and 48.6% in boys; while at 36 months, this rate was 43.3% in girls and 52.1% in boys. The latent class analysis identified two trajectories of weight and length in boys and girls. Girls showed different growth trajectories of weight since 12 months compared with boys.Conclusions Our study reported a relatively low rate of catch-up growth in full-term SGA infants and has identified different growth trajectories of length and weight in boys and girls. We call for attention from health professionals on the growth trajectory of full-term SGA infants to eventually promote their health potentials

Local partial least squares based on global PLS scores

A local‐based method for near‐infrared spectroscopy predictions, the local partial least squares regression on global PLS scores (LPLS‐S), is proposed in this work and compared with the usual local PLS (LPLS) regression approach. LPLS‐S is based on the idea of replacing the original spectra with a global PLS score matrix before using the usual LPLS. This is done with the aim of increasing the speed of the calculations, which can be an important parameter for online applications in particular, especially when implemented on large databases. In this study, the performance of the two local approaches was compared in terms of efficiency and speed. It could be concluded that the root‐mean‐square error of prediction of LPLS and LPLS‐S were 1.1962 and 1.1602, respectively, but the calculation speed for LPLS‐S was more than 20 times faster than for the LPLS algorithm

Inhibition of P‑Glycoprotein Mediated Efflux in Caco‑2 Cells by Phytic Acid

Phytic acid (IP6) is a natural phosphorylated inositol, which is abundantly present in most cereal grains and seeds. This study investigated the effects of IP6 regulation on P-glycoprotein (P-gp) and its potential mechanisms using <i>in situ</i> and <i>in vitro</i> models. The effective permeability of the typical P-gp substrate rhodamine 123 (R123) in colon was significantly increased from (1.69 ± 0.22) × 10^–5 cm/s in the control group to (3.39 ± 0.417) × 10^–5 cm/s (<i>p</i> < 0.01) in the 3.5 mM IP6 group. Additionally, IP6 can concentration-dependently decrease the R123 efflux ratio in both Caco-2 and MDCK II-MDR1 cell monolayers and increase intracellular R123 accumulation in Caco-2 cells. Furthermore, IP6 noncompetitively inhibited P-gp by impacting R123 efflux kinetics. The noncompetitive inhibition of P-gp by IP6 was likely due to decreases in P-gp ATPase activity and P-gp molecular conformational changes induced by IP6. In summary, IP6 is a promising P-gp inhibitor candidate