Language Model Pre-Training with Sparse Latent Typing

Modern large-scale Pre-trained Language Models (PLMs) have achieved tremendous success on a wide range of downstream tasks. However, most of the LM pre-training objectives only focus on text reconstruction, but have not sought to learn latent-level interpretable representations of sentences. In this paper, we manage to push the language models to obtain a deeper understanding of sentences by proposing a new pre-training objective, Sparse Latent Typing, which enables the model to sparsely extract sentence-level keywords with diverse latent types. Experimental results show that our model is able to learn interpretable latent type categories in a self-supervised manner without using any external knowledge. Besides, the language model pre-trained with such an objective also significantly improves Information Extraction related downstream tasks in both supervised and few-shot settings. Our code is publicly available at: https://github.com/renll/SparseLT.Comment: EMNLP 2022 (Oral

Revisiting large-scale interception patterns constrained by a synthesis of global experimental data

Rainfall interception loss remains one of the most uncertain fluxes in the global water balance, hindering water management in forested regions and precluding an accurate formulation in climate models. Here, a synthesis of interception loss data from past field experiments conducted worldwide is performed, resulting in a meta-analysis comprising 166 forest sites and 17 agricultural plots. This meta-analysis is used to constrain a global process-based model driven by satellite-observed vegetation dynamics, potential evaporation and precipitation. The model considers sub-grid heterogeneity and vegetation dynamics and formulates rainfall interception for tall and short vegetation separately. A global, 40-year (1980–2019), 0.1∘ spatial resolution, daily temporal resolution dataset is created, analysed and validated against in situ data. The validation shows a good consistency between the modelled interception and field observations over tall vegetation, both in terms of correlations and bias. While an underestimation is found in short vegetation, the degree to which it responds to in situ representativeness errors and difficulties inherent to the measurement of interception in short vegetated ecosystems is unclear. Global estimates are compared to existing datasets, showing overall comparable patterns. According to our findings, global interception averages to 73.81 mm yr−1 or 10.96 × 103 km3 yr−1, accounting for 10.53 % of continental rainfall and approximately 14.06 % of terrestrial evaporation. The seasonal variability of interception follows the annual cycle of canopy cover, precipitation, and atmospheric demand for water. Tropical rainforests show low intra-annual vegetation variability, and seasonal patterns are dictated by rainfall. Interception shows a strong variance among vegetation types and biomes, supported by both the modelling and the meta-analysis of field data. The global synthesis of field observations and the new global interception dataset will serve as a benchmark for future investigations and facilitate large-scale hydrological and climate research.</p