Self-Evolution Learning for Discriminative Language Model Pretraining

Masked language modeling, widely used in discriminative language model (e.g., BERT) pretraining, commonly adopts a random masking strategy. However, random masking does not consider the importance of the different words in the sentence meaning, where some of them are more worthy to be predicted. Therefore, various masking strategies (e.g., entity-level masking) are proposed, but most of them require expensive prior knowledge and generally train from scratch without reusing existing model weights. In this paper, we present Self-Evolution learning (SE), a simple and effective token masking and learning method to fully and wisely exploit the knowledge from data. SE focuses on learning the informative yet under-explored tokens and adaptively regularizes the training by introducing a novel Token-specific Label Smoothing approach. Experiments on 10 tasks show that our SE brings consistent and significant improvements (+1.43~2.12 average scores) upon different PLMs. In-depth analyses demonstrate that SE improves linguistic knowledge learning and generalization.Comment: Accepted to Findings of ACL202

Revisiting Token Dropping Strategy in Efficient BERT Pretraining

Token dropping is a recently-proposed strategy to speed up the pretraining of masked language models, such as BERT, by skipping the computation of a subset of the input tokens at several middle layers. It can effectively reduce the training time without degrading much performance on downstream tasks. However, we empirically find that token dropping is prone to a semantic loss problem and falls short in handling semantic-intense tasks. Motivated by this, we propose a simple yet effective semantic-consistent learning method (ScTD) to improve the token dropping. ScTD aims to encourage the model to learn how to preserve the semantic information in the representation space. Extensive experiments on 12 tasks show that, with the help of our ScTD, token dropping can achieve consistent and significant performance gains across all task types and model sizes. More encouragingly, ScTD saves up to 57% of pretraining time and brings up to +1.56% average improvement over the vanilla token dropping.Comment: Accepted to ACL2023 Main Conferenc

Low lymphocyte to high-density lipoprotein ratio predicts mortality in sepsis patients

BackgroundThe lymphocyte-to-high-density lipoprotein (HDL) ratio (LHR) is associated with both inflammation and immunity, and may have the potential to predict the prognosis of sepsis. Our study aimed to evaluate the relationship between LHR and sepsis-related mortality.MethodsWe collected data from the Medical Information Mart for Intensive Care IV (MIMIC-IV, version 2.2) database by targeting patients who met the Sepsis-3 criteria and recorded the absolute values of lymphocytes and HDL after admission. We then used restricted cubic splines based on logistic regression to simulate the relationship between the LHR and 90-day mortality. Subsequently, the hazardous threshold was derived based on the mortality curve, and further evaluations were performed using different methods and data sources for hazardous threshold.ResultsWe ultimately included 1027 eligible patients from the MIMIC-IV database and described the nonlinear relationship between LHR and 90-day mortality. Based on the curve, an LHR of ≤ 0.6 indicated harmful threshold, and the odds ratio for mortality was 1.74 (P=0.001). The outperforming hazard was particularly marked in patients with chronic lung disease and remained consistent after adjusting for baseline data and validating multiple data sources.ConclusionsThe LHR has prognostic value in patients with sepsis, and an LHR ≤ 0.6 is a deleterious load that increases mortality

Global Sulfur Dioxide Emissions and the Driving Forces

The presence of sulfur dioxide (SO_{2}) in the air is a global concern because of its severe environmental and public health impacts. Recent evidence from satellite observations shows rapid changes in the spatial distribution of global SO_{2} emissions, but such features are generally missing in global emission inventories that use a bottom-up method due to the lack of up-to-date information, especially in developing countries. Here, we rely on the latest data available on emission activities, control measures, and emission factors to estimate global SO2 emissions for the period 1960–2014 on a 0.1° × 0.1° spatial resolution. We design two counterfactual scenarios to isolate the contributions of emission activity growth and control measure deployment on historical SO_{2} emission changes. We find that activity growth has been the major factor driving global SO_{2} emission changes overall, but control measure deployment is playing an increasingly important role. With effective control measures deployed in developed countries, the predominant emission contributor has shifted from developed countries in the early 1960s (61%) to developing countries at present (83%). Developing countries show divergency in mitigation strategies and thus in SO_{2} emission trends. Stringent controls in China are driving the recent decline in global emissions. A further reduction in SO_{2} emissions would come from a large number of developing nations that currently lack effective SO_{2} emission controls

Birth of rats following nuclear exchange at the 2-cell stage

We report full-term development of nuclear transfer embryos following nuclear exchange at the 2-cell stage. Nuclei from 2-cell rat embryos were transferred into enucleated 2-cell embryos and developed to term after transfer to recipients (NT2). Pronuclear exchange in zygotes was used for comparison (NT1). Zygotes and 2-cell embryos were harvested from 4-week-old female Sprague-Dawley rats. Nuclear transfer was performed by transferring the pronuclei or karyoplasts into the perivitelline space of recipient embryos followed by electrofusion to reconstruct embryos. Fused couplets were cultured for 4 or 24 h before being transferred into day 1 pseudopregnant recipients (Hooded Wistar) at the 1- or 2-cell stage. In vitro culture was also carried out to check the developmental competence of the embryos. In vitro development to the blastocyst stage was not significantly different between the two groups (NT1, 34.3%; NT2, 45.0%). Two of three recipients from NT1 and two of five recipients from NT2 became pregnant. Six pups (3 from NT1, 3 from NT2) were delivered from the four foster mothers. Three female pups survived; 2 from NT1 and 1 from NT2. At 2 months of age these pups appeared healthy, and were mated with Sprague-Dawley males. One rat derived from NT1 delivered 15 pups (5 males, 10 females) as did the rat from NT2 (7 males, 8 females). Our results show that by using karyoplasts from 2-cell stage embryos as nuclear donors and reconstructing them with enucleated 2-cell embryos, healthy rats can be produced

Exchange-biased hybrid gamma-Fe2O3/NiO core-shell nanostructures:three-step synthesis, microstructure, and magnetic properties

A two-step solvothermal method combining a calcination process was conducted to synthesize gamma-Fe2O3/NiO core-shell nanostructures with controlled microstructure. The formation mechanism of this binary system has been discussed, and the influence of microstructures on magnetic properties has been analyzed in detail. Microstructural characterizations reveal that the NiO shells consisted of many irregular nanosheets with disordered orientations and monocrystalline structures, packed on the surface of the gamma-Fe2O3 microspheres. Both the grain size and NiO content of nanostructures increase with the increasing calcination temperature from 300 degrees C to 400 degrees C, accompanied by an enhancement of the compactness of NiO shells. Magnetic studies indicate that their magnetic properties are determined by four factors: the size effect, NiO phase content, interface microstructure, i.e. contact mode, area, roughness and compactness, and FM-AFM (where FM and AFM denote the ferromagnetic gamma-Fe2O3 and the antiferromagnetic NiO components, respectively) coupling effect. At 5 K, the gamma-Fe2O3/NiO core-shell nanostructures display certain exchange bias (H-E = 60 Oe) and enhanced coercivity (H-C = 213 Oe)

Impacts of air pollutants from rural Chinese households under the rapid residential energy transition

Rural residential energy consumption in China is experiencing a rapid transition towards clean energy, nevertheless, solid fuel combustion remains an important emission source. Here we quantitatively evaluate the contribution of rural residential emissions to PM2.5 (particulate matter with an aerodynamic diameter less than 2.5 μm) and the impacts on health and climate. The clean energy transitions result in remarkable reductions in the contributions to ambient PM2.5, avoiding 130,000 (90,000-160,000) premature deaths associated with PM2.5 exposure. The climate forcing associated with this sector declines from 0.057 ± 0.016 W/m2 in 1992 to 0.031 ± 0.008 W/m2 in 2012. Despite this, the large remaining quantities of solid fuels still contributed 14 ± 10 μg/m3 to population-weighted PM2.5 in 2012, which comprises 21 ± 14% of the overall population-weighted PM2.5 from all sources. Rural residential emissions affect not only rural but urban air quality, and the impacts are highly seasonal and location dependent

Polysaccharide from Sepia esculenta ink and cisplatin inhibit synergistically proliferation and metastasis of triple-negative breast cancer MDA-MB-231 cells

Objective(s): This paper aims to investigate synergistic inhibition of polysaccharide from Sepia esculenta ink (SIP), a newly isolated marine polysaccharide in our laboratory, on breast cancer MDA-MB-231 cells exposed to cisplatin. Materials and Methods: Cell viability of MDA-MB-231 cells was determined by CCK 8 assay. Median-effect concentration was analyzed using Chou-Talalay method that was also subjected to determine cell inhibition ratio and combined index, as well as interaction between SIP and cisplatin. Proliferation and migration abilities were detected with plate colony formation assay and cell wound scratch assay, respectively. Expression of MMP-2 and MMP-9 proteins was measured with Western blot assay. Results: Data showed that SIP not only suppressed proliferation and migration of MDA-MB-231 cells, and expression of MMP-2 and MMP-9 proteins, also promoted inhibition of cisplatin on proliferation, migration and MMPs expression of MDA-MB-231 cells, which indicates synergy inhibition of drug combination of SIP and cisplatin on breast cancer cells. The median-effect concentrations of cisplatin and SIP were 4.9 and 1659.6 μg/ml, respectively. Whereas the concentration of combination drug was 158.5 μg/ml. The data indicated that drug combination can decrease dosages of the two single agents, especially the usual dosage of cisplatin. Conclusion: This research demonstrated that SIP repressed proliferation and metastasis of MDA-MB-231 cells and promoted anticancer effect of cisplatin on the breast cancer cells. The data suggested that SIP is a potential natural drug that can be used as an auxiliary medicine alongside chemotherapy in treating breast cancer

Mutations in WNT10B Are Identified in Individuals with Oligodontia

Supplemental Data Supplemental Data include six figures and three tables and can be found with this article online at http://dx.doi.org/10.1016/j.ajhg.2016.05.012. Supplemental Data Document S1. Figures S1–S6 and Tables S1–S3 Download Document S2. Article plus Supplemental Data Download Web Resources Allen Brain Atlas, http://www.brain-map.org/ Eurexpress, http://www.eurexpress.org/ee/ ExAC Browser, http://exac.broadinstitute.org/ GEO Profiles, http://www.ncbi.nlm.nih.gov/geoprofiles HGMD, http://www.biobase-international.com/product/hgmd MutationTaster, http://www.mutationtaster.org/ OMIM, http://www.omim.org RefSeq, http://www.ncbi.nlm.nih.gov/refseq/ Tooth agenesis is one of the most common developmental anomalies in humans. Oligodontia, a severe form of tooth agenesis, is genetically and phenotypically a heterogeneous condition. Although significant efforts have been made, the genetic etiology of dental agenesis remains largely unknown. In the present study, we performed whole-exome sequencing to identify the causative mutations in Chinese families in whom oligodontia segregates with dominant inheritance. We detected a heterozygous missense mutation (c.632G>A [p.Arg211Gln]) in WNT10B in all affected family members. By Sanger sequencing a cohort of 145 unrelated individuals with non-syndromic oligodontia, we identified three additional mutations (c.569C>G [p.Pro190Arg], c.786G>A [p.Trp262∗], and c.851T>G [p.Phe284Cys]). Interestingly, analysis of genotype-phenotype correlations revealed that mutations in WNT10B affect the development of permanent dentition, particularly the lateral incisors. Furthermore, a functional assay demonstrated that each of these mutants could not normally enhance the canonical Wnt signaling in HEPG2 epithelial cells, in which activity of the TOPFlash luciferase reporter was measured. Notably, these mutant WNT10B ligands could not efficiently induce endothelial differentiation of dental pulp stem cells. Our findings provide the identification of autosomal-dominant WNT10B mutations in individuals with oligodontia, which increases the spectrum of congenital tooth agenesis and suggests attenuated Wnt signaling in endothelial differentiation of dental pulp stem cells

Perioperative, oncologic, and functional outcomes of robot-assisted partial nephrectomy for special types of renal tumors (hilar, endophytic, or cystic): an evidence-based analysis of comparative outcomes

PurposeThis study aims to perform a pooled analysis to compare the outcomes of robot-assisted partial nephrectomy (RAPN) between complex tumors (hilar, endophytic, or cystic) and non-complex tumors (nonhilar, exophytic, or solid) and evaluate the effects of renal tumor complexity on outcomes in patients undergoing RAPN.MethodsFour databases were systematically searched, including Science, PubMed, Web of Science, and Cochrane Library, to identify relevant studies published in English up to December 2022. Review Manager 5.4 was used for statistical analyses and calculations. The study was registered with PROSPERO (Registration number: CRD42023394792).ResultsIn total, 14 comparative trials, including 3758 patients were enrolled. Compared to non-complex tumors, complex tumors were associated with a significantly longer warm ischemia time (WMD 3.67 min, 95% CI 1.78, 5.57; p = 0.0001), more blood loss (WMD 22.84 mL, 95% CI 2.31, 43.37; p = 0.03), and a higher rate of major complications (OR 2.35, 95% CI 1.50, 3.67; p = 0.0002). However, no statistically significant differences were found between the two groups in operative time, length of stay, transfusion rates, conversion to open nephrectomy and radical nephrectomy rates, estimated glomerular filtration rate (eGFR) decline, intraoperative complication, overall complication, positive surgical margins (PSM), local recurrence, and trifecta achievement.ConclusionsRAPN can be a safe and effective procedure for complex tumors (hilar, endophytic, or cystic) and provides comparable functional and oncologic outcomes to non-complex tumors.Systematic review registrationhttps://www.crd.york.ac.uk/prospero/display_record.php?RecordID=394792, identifier CRD42023394792