132 research outputs found
Segatron: Segment-Aware Transformer for Language Modeling and Understanding
Transformers are powerful for sequence modeling. Nearly all state-of-the-art
language models and pre-trained language models are based on the Transformer
architecture. However, it distinguishes sequential tokens only with the token
position index. We hypothesize that better contextual representations can be
generated from the Transformer with richer positional information. To verify
this, we propose a segment-aware Transformer (Segatron), by replacing the
original token position encoding with a combined position encoding of
paragraph, sentence, and token. We first introduce the segment-aware mechanism
to Transformer-XL, which is a popular Transformer-based language model with
memory extension and relative position encoding. We find that our method can
further improve the Transformer-XL base model and large model, achieving 17.1
perplexity on the WikiText-103 dataset. We further investigate the pre-training
masked language modeling task with Segatron. Experimental results show that
BERT pre-trained with Segatron (SegaBERT) can outperform BERT with vanilla
Transformer on various NLP tasks, and outperforms RoBERTa on zero-shot sentence
representation learning.Comment: Accepted by AAAI 202
Stratified Rule-Aware Network for Abstract Visual Reasoning
Abstract reasoning refers to the ability to analyze information, discover
rules at an intangible level, and solve problems in innovative ways. Raven's
Progressive Matrices (RPM) test is typically used to examine the capability of
abstract reasoning. The subject is asked to identify the correct choice from
the answer set to fill the missing panel at the bottom right of RPM (e.g., a
33 matrix), following the underlying rules inside the matrix. Recent
studies, taking advantage of Convolutional Neural Networks (CNNs), have
achieved encouraging progress to accomplish the RPM test. However, they partly
ignore necessary inductive biases of RPM solver, such as order sensitivity
within each row/column and incremental rule induction. To address this problem,
in this paper we propose a Stratified Rule-Aware Network (SRAN) to generate the
rule embeddings for two input sequences. Our SRAN learns multiple granularity
rule embeddings at different levels, and incrementally integrates the
stratified embedding flows through a gated fusion module. With the help of
embeddings, a rule similarity metric is applied to guarantee that SRAN can not
only be trained using a tuplet loss but also infer the best answer efficiently.
We further point out the severe defects existing in the popular RAVEN dataset
for RPM test, which prevent from the fair evaluation of the abstract reasoning
ability. To fix the defects, we propose an answer set generation algorithm
called Attribute Bisection Tree (ABT), forming an improved dataset named
Impartial-RAVEN (I-RAVEN for short). Extensive experiments are conducted on
both PGM and I-RAVEN datasets, showing that our SRAN outperforms the
state-of-the-art models by a considerable margin.Comment: AAAI 2021 paper. Code: https://github.com/husheng12345/SRA
Highly sensitive magnetic properties and large linear magnetoresistance in antiferromagnetic CrxSe(0.875\lex\le1)single crystals
CrxSe (x\le1) is a class of quasi-layered binary compounds with potential
applications in spintronics due to its intriguing antiferromagnetic properties.
In this work, CrxSe single crystals with high Cr content (x=0.87, 0.91 and
0.95) were grown, and their magnetic and transport properties were investigated
in detail. It is found that with small increase of Cr content, the N\'eel
temperature (TN) of the samples can dramatically increase from 147 K to 257 K,
accompanied with obvious changes in the magnetic anisotropy and hysteresis. The
phenomena of field-induced spin-flop transitions were unveiled in these alloys,
indicating their comparatively low anisotropy. The magnetoresistance (MR) of
the three compounds showed positive dependence at low temperatures and
particularly, non-saturated linear positive MR was observed in Cr0.91Se and
Cr0.95Se, with a large value of 16.2% achieved in Cr0.91Se (10K, 9T). The
calculated Fermi surface and MR showed that the quasi-linear MR is a product of
carrier compensation. Our work revealed highly sensitive magnetic and transport
properties in the Cr-Se compounds, which can lay foundation when constructing
further antiferromagnetic spintronic devices based on them
Growth of high-quality CrI3 single crystals and engineering of its magnetic properties via V and Mn doping
CrI3, as a soft van der Waals layered magnetic material, has been widely
concerned and explored for its magnetic complexity and tunability. In this
work, high quality and large size thin CrI3, V and Mn doped single crystals
were prepared by chemical vapor transfer method. A remarkable irreversible
Barkhausen effect was observed in CrI3 and CrMn0.06I3, which can be attributed
to the low dislocation density that facilitates movement of the domain walls.
In addition, the introduction of the doping element Mn allows higher saturation
magnetization intensity. Cr0.5V0.5I3 exhibits substantially increased
coercivity force and larger magnetocrystalline anisotropy compared to CrI3,
while kept similar Curie temperature and good environmental stability. The
first principles calculations suggest direct and narrowed band gaps in
Cr0.5V0.5I3 and VI3 comparing to CrI3. The smaller band gaps and good hard
magnetic property make Cr0.5V0.5I3 an alternative choice to future research of
spintronic devices
Transforming Research on Recreational Ecosystem Services into Applications and Governance
The science-practice gap has recently been discussed as a critical challenge restricting sustainable growth and development in all facets of our society, including explorations of Recreation Ecosystem Services (RES). To better explore how well the scientific study of RES and its application are connected, this paper aims to synthesize empirical evidence based on an in-depth and systematic literature review. We found that studies of RES have not effectively transformed into the decision-making and long-term planning of our cities. From 2005 to 2020, only 13% of studies referred to specific applications, and about 40% of papers mentioned no applications or practical implications for their research. However, RES research has many potential applications, which can be categorised into six main aspects. In terms of non-spatial improvement: Improved monetary benefits (40%), non-monetary benefits (30%); in terms of spatial improvement: space with high recreational potential or degradation (7%), the relation between supply and demand (7%); and Cross-service governance (16%). After combining the results of various studies, we developed a framework starting from applicable problems and their solutions, which can incorporate the outcomes of RES research while systematically narrowing down the research questions and methods. The framework offers a starting point for further research that can modify and improve in bridging science-practice gaps in RES studies.National Natural Science Foundation of ChinaPeer Reviewe
Multi-Compartment Electric Vehicle Routing Problem for Perishable Products
The study first proposes a heterogeneous fleet, multi-compartment electric vehicle routing problem for perishable products (MCEVRP-PP). We capture a lot of practical demands and constraints of the MCEVRP-PP, such as multiple temperature zones, the hard time window, charging more than once during delivery, various power consumption per unit of refrigeration, etc. We model the MCEVRP-PP as a mixed integer program and aim to optimize the total cost including vehicle fixed cost, power cost, and cooling cost. A hybrid ant colony optimization (HACO) is developed to solve the problem. In the transfer rule, the time window is introduced to improve flexibility in route construction. According to the features of multi-compartment electric vehicles, the capacity constraint judgment algorithm is developed in route construction. Six local search strategies are designed with time windows, recharging stations, etc. Experiments based on various instances validate that HACO solves MCEVRP-PP more effectively than the ant colony optimization (ACO). Compared with fuel vehicles and single-compartment vehicles, electric vehicles and multi-compartment electric vehicles can save the total cost and mileage, and increase utilization of vehicles
Vanishing of the anomalous Hall effect and enhanced carrier mobility in the spin-gapless ferromagnetic Mn2CoGa1-xAlx alloys
Spin gapless semiconductor (SGS) has attracted long attention since its
theoretical prediction, while concrete experimental hints are still lack in the
relevant Heusler alloys. Here in this work, by preparing the series alloys of
Mn2CoGa1-xAlx (x=0, 0.25, 0.5, 0.75 and 1), we identified the vanishing of
anomalous Hall effect in the ferromagnetic Mn2CoGa (or x=0.25) alloy in a wide
temperature interval, accompanying with growing contribution from the ordinary
Hall effect. As a result, comparatively low carrier density (1020 cm-3) and
high carrier mobility (150 cm2/Vs) are obtained in Mn2CoGa (or x=0.25) alloy in
the temperature range of 10-200K. These also lead to a large dip in the related
magnetoresistance at low fields. While in high Al content, despite the
magnetization behavior is not altered significantly, the Hall resistivity is
instead dominated by the anomalous one, just analogous to that widely reported
in Mn2CoAl. The distinct electrical transport behavior of x=0 and x=0.75 (or 1)
is presently understood by their possible different scattering mechanism of the
anomalous Hall effect due to the differences in atomic order and conductivity.
Our work can expand the existing understanding of the SGS properties and offer
a better SGS candidate with higher carrier mobility that can facilitate the
application in the spin-injected related devices
Genome-wide analysis reveals four key transcription factors associated with cadmium stress in creeping bentgrass (Agrostis stolonifera L.)
Cadmium (Cd) toxicity seriously affects the growth and development of plants, so studies on uptake, translocation, and accumulation of Cd in plants are crucial for phytoremediation. However, the molecular mechanism of the plant response to Cd stress remains poorly understood. The main objective of this study was to reveal differentially expressed genes (DEGs) under lower (BT2_5) and higher (BT43) Cd concentration treatments in creeping bentgrass. A total of 463,184 unigenes were obtained from creeping bentgrass leaves using RNA sequencing technology. Observation of leaf tissue morphology showed that the higher Cd concentration damages leaf tissues. Four key transcription factor (TF) families, WRKY, bZIP, ERF, and MYB, are associated with Cd stress in creeping bentgrass. Our findings revealed that these four TFs play crucial roles during the creeping bentgrass response to Cd stress. This study is mainly focused on the molecular characteristics of DEGs under Cd stress using transcriptomic analysis in creeping bentgrass. These results provide novel insight into the regulatory mechanisms of respond to Cd stress and enrich information for phytoremediation
Nutritional Interventions Improved Rumen Functions and Promoted Compensatory Growth of Growth-Retarded Yaks as Revealed by Integrated Transcripts and Microbiome Analyses
Growth retardation reduces the incomes of livestock farming. However, effective nutritional interventions to promote compensatory growth and the mechanisms involving digestive tract microbiomes and transcripts have yet to be elucidated. In this study, Qinghai plateau yaks, which frequently suffer from growth retardation due to malnutrition, were used as an experimental model. Young growth-retarded yaks were pastured (GRP), fed basal ration (GRB), fed basal ration addition cysteamine hydrochloride (CSH; GRBC) or active dry yeast (ADY; GRBY). Another group of growth normal yak was pastured as a positive control (GNP). After 60-day nutritional interventions, the results showed that the average daily gain (ADG) of GRB was similar to the level of GNP, and the growth rates of GRBC and GRBY were significantly higher than the level of GNP (P < 0.05). Basal rations addition of CSH or ADY either improved the serum biochemical indexes, decreased serum LPS concentration, facilitated ruminal epithelium development and volatile fatty acids (VFA) fermentation of growth-retarded yaks. Comparative transcriptome in rumen epithelium between growth-retarded and normal yaks identified the differentially expressed genes mainly enriched in immune system, digestive system, extracellular matrix and cell adhesion pathways. CSH addition and ADY addition in basal rations upregulated ruminal VFA absorption (SLC26A3, PAT1, MCT1) and cell junction (CLDN1, CDH1, OCLN) gene expression, and downregulated complement system (C2, C7) gene expression in the growth-retarded yaks. 16S rDNA results showed that CSH addition and ADY addition in basal rations increased the rumen beneficial bacterial populations (Prevotella_1, Butyrivibrio_2, Fibrobacter) of growth-retarded yaks. The correlation analysis identified that ruminal VFAs and beneficial bacteria abundance were significantly positively correlated with cell junction and VFA absorption gene expressions and negatively correlated with complement system gene expressions on the ruminal epithelium. Therefore, CSH addition and ADY addition in basal rations promoted rumen health and body growth of growth-retarded yaks, of which basal ration addition of ADY had the optimal growth-promoting effects. These results suggested that improving nutrition and probiotics addition is a more effective method to improve growth retardation caused by gastrointestinal function deficiencies
- …