Segatron: Segment-Aware Transformer for Language Modeling and Understanding

Transformers are powerful for sequence modeling. Nearly all state-of-the-art language models and pre-trained language models are based on the Transformer architecture. However, it distinguishes sequential tokens only with the token position index. We hypothesize that better contextual representations can be generated from the Transformer with richer positional information. To verify this, we propose a segment-aware Transformer (Segatron), by replacing the original token position encoding with a combined position encoding of paragraph, sentence, and token. We first introduce the segment-aware mechanism to Transformer-XL, which is a popular Transformer-based language model with memory extension and relative position encoding. We find that our method can further improve the Transformer-XL base model and large model, achieving 17.1 perplexity on the WikiText-103 dataset. We further investigate the pre-training masked language modeling task with Segatron. Experimental results show that BERT pre-trained with Segatron (SegaBERT) can outperform BERT with vanilla Transformer on various NLP tasks, and outperforms RoBERTa on zero-shot sentence representation learning.Comment: Accepted by AAAI 202

3-[(R)-3,3-Dichloro-2-hydroxy­prop­yl]-8-hydr­oxy-6-meth­oxy-1H-isochromen-1-one

The title compound, C13H12Cl2O5, is an isocoumarin compound which has been isolated from the ethyl acetate extract of the fermentation broth of actinomycete Streptomyces sp. (V4) from the South China Sea. There are intra- and inter­molecular hydrogen bonds and halogen bonds [Cl⋯Cl = 3.434 (2) Å; C—Cl⋯Cl = 121.6°]. The intermolecular O—H⋯O hydrogen bonds link mol­ecules into chains along the b axis

Elevation of High-Mobility Group Protein Box-1 in Serum Correlates with Severity of Acute Intracerebral Hemorrhage

High-mobility group protein box-1 (HMGB1) is a proinflammatory involved in many inflammatory diseases. However, its roles in intracerebral hemorrhage (ICH) remain unknown. The purpose of this study was to examine the correlation between changes in serum levels of HMGB1 following acute ICH and the severity of stroke as well as the underlying mechanism. Changes in serum levels of HMGB1 in 60 consecutive patients with primary hemispheric ICH within 12 hours of onset of symptoms were determined. The correlation of HMGB1 with disease severity, IL-6, and TNF-α was analyzed. Changes in HMGB1 levels were detected with ELISA and Western blot. Compared with normal controls, patients with ICH had markedly elevated levels of HMGB1, which was significantly correlated with the levels of IL-6 and TNF-α, NIHSS score at the 10th day, and mRS score at 3 months. In comparison with the control group, the levels of HMGB1 in the perihematomal tissue in mice with ICH increased dramatically, peaked at 72 hours, and decreased at 5 days. Meanwhile, heme could stimulate cultured microglia to release large amounts of HMGB1 whereas Fe2+/3+ ions failed to stimulate HMGB1 production from microglia. Our findings suggest that HMGB1 may play an essential role in the ICH-caused inflammatory injury

Generation of acetyllysine antibodies and affinity enrichment of acetylated peptides

Lysine acetylation has emerged as one of the major post-translational modifications, as indicated by its roles in chromatin remodeling, activation of transcription factors and, most recently, regulation of metabolic enzymes. Identification of acetylation sites in a protein is the first essential step for functional characterization of acetylation in physiological regulation. However, the study of the acetylome is hindered by the lack of suitable physical and biochemical properties of the acetyl group and existence of high-abundance acetylated histones in the cell, and needs a robust method to overcome these problems. Here we present protocols for (i) using chemically acetylated ovalbumin and synthetic acetylated peptide to generate a pan-acetyllysine antibody and a site-specific antibody to Lys288-acetylated argininosuccinate lyase, respectively; (ii) using subcellular fractionation to reduce highly abundant acetylated histones; and (iii) using acetyllysine antibody affinity purification and mass spectrometry to characterize acetylome of human liver tissue. The entire characterization procedure takes ~2–3 d to complete