Segatron: Segment-Aware Transformer for Language Modeling and Understanding

Transformers are powerful for sequence modeling. Nearly all state-of-the-art language models and pre-trained language models are based on the Transformer architecture. However, it distinguishes sequential tokens only with the token position index. We hypothesize that better contextual representations can be generated from the Transformer with richer positional information. To verify this, we propose a segment-aware Transformer (Segatron), by replacing the original token position encoding with a combined position encoding of paragraph, sentence, and token. We first introduce the segment-aware mechanism to Transformer-XL, which is a popular Transformer-based language model with memory extension and relative position encoding. We find that our method can further improve the Transformer-XL base model and large model, achieving 17.1 perplexity on the WikiText-103 dataset. We further investigate the pre-training masked language modeling task with Segatron. Experimental results show that BERT pre-trained with Segatron (SegaBERT) can outperform BERT with vanilla Transformer on various NLP tasks, and outperforms RoBERTa on zero-shot sentence representation learning.Comment: Accepted by AAAI 202

Stratified Rule-Aware Network for Abstract Visual Reasoning

Abstract reasoning refers to the ability to analyze information, discover rules at an intangible level, and solve problems in innovative ways. Raven's Progressive Matrices (RPM) test is typically used to examine the capability of abstract reasoning. The subject is asked to identify the correct choice from the answer set to fill the missing panel at the bottom right of RPM (e.g., a 3$\times$3 matrix), following the underlying rules inside the matrix. Recent studies, taking advantage of Convolutional Neural Networks (CNNs), have achieved encouraging progress to accomplish the RPM test. However, they partly ignore necessary inductive biases of RPM solver, such as order sensitivity within each row/column and incremental rule induction. To address this problem, in this paper we propose a Stratified Rule-Aware Network (SRAN) to generate the rule embeddings for two input sequences. Our SRAN learns multiple granularity rule embeddings at different levels, and incrementally integrates the stratified embedding flows through a gated fusion module. With the help of embeddings, a rule similarity metric is applied to guarantee that SRAN can not only be trained using a tuplet loss but also infer the best answer efficiently. We further point out the severe defects existing in the popular RAVEN dataset for RPM test, which prevent from the fair evaluation of the abstract reasoning ability. To fix the defects, we propose an answer set generation algorithm called Attribute Bisection Tree (ABT), forming an improved dataset named Impartial-RAVEN (I-RAVEN for short). Extensive experiments are conducted on both PGM and I-RAVEN datasets, showing that our SRAN outperforms the state-of-the-art models by a considerable margin.Comment: AAAI 2021 paper. Code: https://github.com/husheng12345/SRA

Highly sensitive magnetic properties and large linear magnetoresistance in antiferromagnetic CrxSe(0.875\lex\le1)single crystals

CrxSe (x\le1) is a class of quasi-layered binary compounds with potential applications in spintronics due to its intriguing antiferromagnetic properties. In this work, CrxSe single crystals with high Cr content (x=0.87, 0.91 and 0.95) were grown, and their magnetic and transport properties were investigated in detail. It is found that with small increase of Cr content, the N\'eel temperature (TN) of the samples can dramatically increase from 147 K to 257 K, accompanied with obvious changes in the magnetic anisotropy and hysteresis. The phenomena of field-induced spin-flop transitions were unveiled in these alloys, indicating their comparatively low anisotropy. The magnetoresistance (MR) of the three compounds showed positive dependence at low temperatures and particularly, non-saturated linear positive MR was observed in Cr0.91Se and Cr0.95Se, with a large value of 16.2% achieved in Cr0.91Se (10K, 9T). The calculated Fermi surface and MR showed that the quasi-linear MR is a product of carrier compensation. Our work revealed highly sensitive magnetic and transport properties in the Cr-Se compounds, which can lay foundation when constructing further antiferromagnetic spintronic devices based on them

Growth of high-quality CrI3 single crystals and engineering of its magnetic properties via V and Mn doping

CrI3, as a soft van der Waals layered magnetic material, has been widely concerned and explored for its magnetic complexity and tunability. In this work, high quality and large size thin CrI3, V and Mn doped single crystals were prepared by chemical vapor transfer method. A remarkable irreversible Barkhausen effect was observed in CrI3 and CrMn0.06I3, which can be attributed to the low dislocation density that facilitates movement of the domain walls. In addition, the introduction of the doping element Mn allows higher saturation magnetization intensity. Cr0.5V0.5I3 exhibits substantially increased coercivity force and larger magnetocrystalline anisotropy compared to CrI3, while kept similar Curie temperature and good environmental stability. The first principles calculations suggest direct and narrowed band gaps in Cr0.5V0.5I3 and VI3 comparing to CrI3. The smaller band gaps and good hard magnetic property make Cr0.5V0.5I3 an alternative choice to future research of spintronic devices

Transforming Research on Recreational Ecosystem Services into Applications and Governance

The science-practice gap has recently been discussed as a critical challenge restricting sustainable growth and development in all facets of our society, including explorations of Recreation Ecosystem Services (RES). To better explore how well the scientific study of RES and its application are connected, this paper aims to synthesize empirical evidence based on an in-depth and systematic literature review. We found that studies of RES have not effectively transformed into the decision-making and long-term planning of our cities. From 2005 to 2020, only 13% of studies referred to specific applications, and about 40% of papers mentioned no applications or practical implications for their research. However, RES research has many potential applications, which can be categorised into six main aspects. In terms of non-spatial improvement: Improved monetary benefits (40%), non-monetary benefits (30%); in terms of spatial improvement: space with high recreational potential or degradation (7%), the relation between supply and demand (7%); and Cross-service governance (16%). After combining the results of various studies, we developed a framework starting from applicable problems and their solutions, which can incorporate the outcomes of RES research while systematically narrowing down the research questions and methods. The framework offers a starting point for further research that can modify and improve in bridging science-practice gaps in RES studies.National Natural Science Foundation of ChinaPeer Reviewe

Vanishing of the anomalous Hall effect and enhanced carrier mobility in the spin-gapless ferromagnetic Mn2CoGa1-xAlx alloys

Spin gapless semiconductor (SGS) has attracted long attention since its theoretical prediction, while concrete experimental hints are still lack in the relevant Heusler alloys. Here in this work, by preparing the series alloys of Mn2CoGa1-xAlx (x=0, 0.25, 0.5, 0.75 and 1), we identified the vanishing of anomalous Hall effect in the ferromagnetic Mn2CoGa (or x=0.25) alloy in a wide temperature interval, accompanying with growing contribution from the ordinary Hall effect. As a result, comparatively low carrier density (1020 cm-3) and high carrier mobility (150 cm2/Vs) are obtained in Mn2CoGa (or x=0.25) alloy in the temperature range of 10-200K. These also lead to a large dip in the related magnetoresistance at low fields. While in high Al content, despite the magnetization behavior is not altered significantly, the Hall resistivity is instead dominated by the anomalous one, just analogous to that widely reported in Mn2CoAl. The distinct electrical transport behavior of x=0 and x=0.75 (or 1) is presently understood by their possible different scattering mechanism of the anomalous Hall effect due to the differences in atomic order and conductivity. Our work can expand the existing understanding of the SGS properties and offer a better SGS candidate with higher carrier mobility that can facilitate the application in the spin-injected related devices

Genome-wide analysis reveals four key transcription factors associated with cadmium stress in creeping bentgrass (Agrostis stolonifera L.)

Cadmium (Cd) toxicity seriously affects the growth and development of plants, so studies on uptake, translocation, and accumulation of Cd in plants are crucial for phytoremediation. However, the molecular mechanism of the plant response to Cd stress remains poorly understood. The main objective of this study was to reveal differentially expressed genes (DEGs) under lower (BT2_5) and higher (BT43) Cd concentration treatments in creeping bentgrass. A total of 463,184 unigenes were obtained from creeping bentgrass leaves using RNA sequencing technology. Observation of leaf tissue morphology showed that the higher Cd concentration damages leaf tissues. Four key transcription factor (TF) families, WRKY, bZIP, ERF, and MYB, are associated with Cd stress in creeping bentgrass. Our findings revealed that these four TFs play crucial roles during the creeping bentgrass response to Cd stress. This study is mainly focused on the molecular characteristics of DEGs under Cd stress using transcriptomic analysis in creeping bentgrass. These results provide novel insight into the regulatory mechanisms of respond to Cd stress and enrich information for phytoremediation

Nutritional Interventions Improved Rumen Functions and Promoted Compensatory Growth of Growth-Retarded Yaks as Revealed by Integrated Transcripts and Microbiome Analyses

Growth retardation reduces the incomes of livestock farming. However, effective nutritional interventions to promote compensatory growth and the mechanisms involving digestive tract microbiomes and transcripts have yet to be elucidated. In this study, Qinghai plateau yaks, which frequently suffer from growth retardation due to malnutrition, were used as an experimental model. Young growth-retarded yaks were pastured (GRP), fed basal ration (GRB), fed basal ration addition cysteamine hydrochloride (CSH; GRBC) or active dry yeast (ADY; GRBY). Another group of growth normal yak was pastured as a positive control (GNP). After 60-day nutritional interventions, the results showed that the average daily gain (ADG) of GRB was similar to the level of GNP, and the growth rates of GRBC and GRBY were significantly higher than the level of GNP (P < 0.05). Basal rations addition of CSH or ADY either improved the serum biochemical indexes, decreased serum LPS concentration, facilitated ruminal epithelium development and volatile fatty acids (VFA) fermentation of growth-retarded yaks. Comparative transcriptome in rumen epithelium between growth-retarded and normal yaks identified the differentially expressed genes mainly enriched in immune system, digestive system, extracellular matrix and cell adhesion pathways. CSH addition and ADY addition in basal rations upregulated ruminal VFA absorption (SLC26A3, PAT1, MCT1) and cell junction (CLDN1, CDH1, OCLN) gene expression, and downregulated complement system (C2, C7) gene expression in the growth-retarded yaks. 16S rDNA results showed that CSH addition and ADY addition in basal rations increased the rumen beneficial bacterial populations (Prevotella_1, Butyrivibrio_2, Fibrobacter) of growth-retarded yaks. The correlation analysis identified that ruminal VFAs and beneficial bacteria abundance were significantly positively correlated with cell junction and VFA absorption gene expressions and negatively correlated with complement system gene expressions on the ruminal epithelium. Therefore, CSH addition and ADY addition in basal rations promoted rumen health and body growth of growth-retarded yaks, of which basal ration addition of ADY had the optimal growth-promoting effects. These results suggested that improving nutrition and probiotics addition is a more effective method to improve growth retardation caused by gastrointestinal function deficiencies