A Simple yet Effective Self-Debiasing Framework for Transformer Models

Current Transformer-based natural language understanding (NLU) models heavily rely on dataset biases, while failing to handle real-world out-of-distribution (OOD) instances. Many methods have been proposed to deal with this issue, but they ignore the fact that the features learned in different layers of Transformer-based NLU models are different. In this paper, we first conduct preliminary studies to obtain two conclusions: 1) both low- and high-layer sentence representations encode common biased features during training; 2) the low-layer sentence representations encode fewer unbiased features than the highlayer ones. Based on these conclusions, we propose a simple yet effective self-debiasing framework for Transformer-based NLU models. Concretely, we first stack a classifier on a selected low layer. Then, we introduce a residual connection that feeds the low-layer sentence representation to the top-layer classifier. In this way, the top-layer sentence representation will be trained to ignore the common biased features encoded by the low-layer sentence representation and focus on task-relevant unbiased features. During inference, we remove the residual connection and directly use the top-layer sentence representation to make predictions. Extensive experiments and indepth analyses on NLU tasks show that our framework performs better than several competitive baselines, achieving a new SOTA on all OOD test sets

An endoglucanase, GsCelA, from Geobacilus sp. undergoes an intriguing self- truncation process for enhancing activity and thermostability

An endoglucanase, GsCelA, was isolated and cloned from a thermophilic Geobacillus sp. 70PC53 grown in a rice straw compost in southern Taiwan. It was observed that highly purified GsCelA was able to self-truncate, removing a segment of 53 amino acid residues from its C-terminus. The purified GsCelA does not possess any protease activity and this self-truncation process is insensitive to standard protease inhibitors except EDTA and EGTA. This unique self-truncation process takes place at a temperature higher than 10C with an optimal pH between 6-7, and can be further enhanced with certain divalent ions such as Ca+2 and Mg+2. Crystal structure of GsCelA has a typical TIM-barrel configuration with 8 alpha-helices and 8 beta-strands, but with the presence of a divalent ion. Mutations of amino acids residues surrounding this metal ion do not affect the self-truncation process, but some of these mutants have enhanced enzymatic activities. Mutation of the cleavage site between K315 and G316 does not affect the self-truncation process. However, a deletion of ten amino acids near the cleavage site, i.e. from amino acid 310 to 320, slows down the truncation process but does not block it, and a truncated form around 315 amino acids in length eventually appears. This intriguing observation indicates that the self-truncation process is not site specific, but capable of measuring 315 amino acids from the N-terminus as the cleavage site. This self-truncation process also occurs in the native host of this enzyme, Geobacillus sp. 70PC53, with almost all secreted form of this enzyme being self-truncated. The 53 amino-acid-long C-terminal segment removed by this self-truncation process has binding affinity toward both crystal and amorphous cellulose as well as the s cell walls, yet its sequence bears no apparent homology to any known carbohydrate binding motifs. Various other mutation analyses and the structure-based recombination process, SCHEMA, have been carried out, and both the activity and thermostabilty of this enzyme are further improved. The truncated and improved GsCelA has almost twice the activity as the un-truncated form, and its thermostability is also further enhanced with T50 reaching 86C and TA50 higher than 100C, making this enzyme extremely useful in industrial processes carried out at high temperatures, such as the pre-treatment of cellulosic animal feeds during the final drying step. This research was supported by grants from Taiwan Ministry of Science and Technology and from Academia Sinica

Positive Solutions for Sturm-Liouville Boundary Value Problems in a Banach Space

We consider the existence of single and multiple positive solutions for a second-order SturmLiouville boundary value problem in a Banach space. The sufficient condition for the existence of positive solution is obtained by the fixed point theorem of strict set contraction operators in the frame of the ODE technique. Our results significantly extend and improve many known results including singular and nonsingular cases

Enhancement of activity and thermostability of a Geobacillus endoglucanase via a unique self-truncation process

The complete utilization of lignocellulosic biomass requires the hydrolysis of cellulose fibers via the synergistic action of three enzymes: exoglucanase, endoglucanase and beta-glucosidase. GsCelA is a 368-amino-acid endoglucanase secreted from a thermophilic Geobacillus sp. 70PC53 that was isolated form a rice straw compost in south Taiwan. GsCelA belongs to the glycosyl hydrolase family 5 and has a typical TIM barrel structure. This enzyme has excellent lignocellulolytic activity and high thermostability, with optimal temperature at 60℃ and pH at 5.0. The purified GsCelA is capable of carrying out a unique self-truncation process at temperature higher than 10 ℃ with optimal pH at 6-7. This self-truncation process is not due to the action of contaminating proteases and it can be suppressed by EDTA and EGTA, and enhanced by divalent metal ions. This self-truncation process also takes place in vivo in Geobacillus sp. 70PC53. The spontaneous or engineered C-terminal truncation up to 60 amino acids from the C-terminus improves GsCelA specific activity and renders the enzyme more thermostable. To investigate the importance of specific amino acids on the enzymatic activity of GsCelA, site-directed mutagenesis and protein engineering approach were employed to alter amino acid residues unique to this enzyme. It was demonstrated that point mutations Y195T , D55S, G288T and D289L replacements increase the activity of this enzyme by 30%