Knowledge-in-Context: Towards Knowledgeable Semi-Parametric Language Models

Fully-parametric language models generally require a huge number of model parameters to store the necessary knowledge for solving multiple natural language tasks in zero/few-shot settings. In addition, it is hard to adapt to the evolving world knowledge without the costly model re-training. In this paper, we develop a novel semi-parametric language model architecture, Knowledge-in-Context (KiC), which empowers a parametric text-to-text language model with a knowledge-rich external memory. Specifically, the external memory contains six different types of knowledge: entity, dictionary, commonsense, event, script, and causality knowledge. For each input instance, the KiC model adaptively selects a knowledge type and retrieves the most helpful pieces of knowledge. The input instance along with its knowledge augmentation is fed into a text-to-text model (e.g., T5) to generate the output answer, where both the input and the output are in natural language forms after prompting. Interestingly, we find that KiC can be identified as a special mixture-of-experts (MoE) model, where the knowledge selector plays the role of a router that is used to determine the sequence-to-expert assignment in MoE. This key observation inspires us to develop a novel algorithm for training KiC with an instance-adaptive knowledge selector. As a knowledge-rich semi-parametric language model, KiC only needs a much smaller parametric part to achieve superior zero-shot performance on unseen tasks. By evaluating on 40+ different tasks, we show that KiC_Large with 770M parameters easily outperforms large language models (LMs) that are 4-39x larger by a large margin. We also demonstrate that KiC exhibits emergent abilities at a much smaller model scale compared to the fully-parametric models

Thrust: Adaptively Propels Large Language Models with External Knowledge

Although large-scale pre-trained language models (PTLMs) are shown to encode rich knowledge in their model parameters, the inherent knowledge in PTLMs can be opaque or static, making external knowledge necessary. However, the existing information retrieval techniques could be costly and may even introduce noisy and sometimes misleading knowledge. To address these challenges, we propose the instance-level adaptive propulsion of external knowledge (IAPEK), where we only conduct the retrieval when necessary. To achieve this goal, we propose measuring whether a PTLM contains enough knowledge to solve an instance with a novel metric, Thrust, which leverages the representation distribution of a small number of seen instances. Extensive experiments demonstrate that thrust is a good measurement of PTLM models' instance-level knowledgeability. Moreover, we can achieve significantly higher cost-efficiency with the Thrust score as the retrieval indicator than the naive usage of external knowledge on 88% of the evaluated tasks with 26% average performance improvement. Such findings shed light on the real-world practice of knowledge-enhanced LMs with a limited knowledge-seeking budget due to computation latency or costs.Comment: 13 pages, 6 figure

UniBrain: Universal Brain MRI Diagnosis with Hierarchical Knowledge-enhanced Pre-training

Magnetic resonance imaging~(MRI) have played a crucial role in brain disease diagnosis, with which a range of computer-aided artificial intelligence methods have been proposed. However, the early explorations usually focus on the limited types of brain diseases in one study and train the model on the data in a small scale, yielding the bottleneck of generalization. Towards a more effective and scalable paradigm, we propose a hierarchical knowledge-enhanced pre-training framework for the universal brain MRI diagnosis, termed as UniBrain. Specifically, UniBrain leverages a large-scale dataset of 24,770 imaging-report pairs from routine diagnostics. Different from previous pre-training techniques for the unitary vision or textual feature, or with the brute-force alignment between vision and language information, we leverage the unique characteristic of report information in different granularity to build a hierarchical alignment mechanism, which strengthens the efficiency in feature learning. Our UniBrain is validated on three real world datasets with severe class imbalance and the public BraTS2019 dataset. It not only consistently outperforms all state-of-the-art diagnostic methods by a large margin and provides a superior grounding performance but also shows comparable performance compared to expert radiologists on certain disease types

Antibiotic-Induced Primary Biles Inhibit SARS-CoV-2 Endoribonuclease Nsp15 Activity in Mouse Gut

The gut microbiome profile of COVID-19 patients was found to correlate with a viral load of SARS-CoV-2, COVID-19 severity, and dysfunctional immune responses, suggesting that gut microbiota may be involved in anti-infection. In order to investigate the role of gut microbiota in anti-infection against SARS-CoV-2, we established a high-throughput in vitro screening system for COVID-19 therapeutics by targeting the endoribonuclease (Nsp15). We also evaluated the activity inhibition of the target by substances of intestinal origin, using a mouse model in an attempt to explore the interactions between gut microbiota and SARS-CoV-2. The results unexpectedly revealed that antibiotic treatment induced the appearance of substances with Nsp15 activity inhibition in the intestine of mice. Comprehensive analysis based on functional profiling of the fecal metagenomes and endoribonuclease assay of antibiotic-enriched bacteria and metabolites demonstrated that the Nsp15 inhibitors were the primary bile acids that accumulated in the gut as a result of antibiotic-induced deficiency of bile acid metabolizing microbes. This study provides a new perspective on the development of COVID-19 therapeutics using primary bile acids

Efficient generation of human primordial germ cell-like cells from pluripotent stem cells in a methylcellulose-based 3D system at large scale

Background The mechanisms underlying human germ cell development and infertility remain largely unknown due to bioethical issues and the shortage of experimental materials. Therefore, an effective in vitro induction system of human primordial germ-like cells (hPGCLCs) from human pluripotent stem cells (hPSC) is in high demand. The current strategies used for the generation of hPGCLCs are not only costly but also difficult to perform at a large scale, thereby posing barriers to further research. In this study, we attempted to solve these problems by providing a new 3D culture system for hPGCLC differentiation. Methods The efficiency and relative yield of a methylcellulose (MC)-based 3D hPGCLC induction system were first compared with that of a conventional U96 system. Then, we examined the gene expression of germ cell marker genes and the key epigenetic modifications of the EpCAM-/INTEGRINα6-high cells from the 3D MC induction system and the U96 system via quantitative PCR and immunofluorescence. Finally, the reliability of the MC-based 3D hPGCLC induction system was evaluated via the generation of induced pluripotent stem cells (iPSCs) from the testicular cells of one patient with obstructive azoospermia (OA) and followed by the subsequent differentiation of iPSCs into the germ cell lineage. Results In the present study, we demonstrated that the 3D MC induction system combined with low-cell attachment plates facilitated the generation of hPGCLCs at a large scale. We found that the hPGCLCs generated via the MC system shared similar characteristics to that via the U96 system in terms of the gene expression profiles, germ cell-specific markers, epigenetic modification states and cellular states. In addition, hPGCLCs from iPSCs derived from one OA patient were generated with high efficiency via the present 3D MC induction system. Discussion The in vitro induction of hPGCLCs from human embryonic stem cells (hESCs)/human induced pluripotent stem cells (hiPSCs) has significant implications in exploring the underlying mechanisms of the origin and specification of hPGCs and the epigenetic programming of the human germ line as well as treating male infertility. Here, we developed a simple and efficient 3D induction system to generate hPGCLCs from hESCs/hiPSCs at a large scale, which facilitated the study of human germ cell development and stem cell-based reproductive medicine

Exploring the Influence Mechanism of Meteorological Conditions on the Concentration of Suspended Solids and Chlorophyll-a in Large Estuaries Based on MODIS Imagery

In estuary areas, meteorological conditions have become unstable under the continuous effects of climate change, and the ecological backgrounds of such areas have strongly been influenced by anthropic activities. Consequently, the water quality of these areas is obviously affected. In this research, we identified periods of fluctuation of the general meteorological conditions in the Yangtze River Estuary using a wavelet analysis. Additionally, we performed a spatiotemporal evaluation of the water quality in the fluctuating period by using remote sensing modeling. Then, we explored how the fluctuating meteorological factors affect the distribution of total suspended solids (TSS) and chlorophyll-a (Chla) concentration. (1) The results show that from 2000 to 2015, temperature did not present significant fluctuations, while wind speed (WS) and precipitation (PR) presented the same fluctuation period from January 2012 to December 2012. (2) Based on the measured water sample data associated with Moderate Resolution Imaging Spectroradiometer (MODIS) imagery, we developed a water quality algorithm and depicted the TSS and Chla concentrations within the WS and PR fluctuating period. (3) We found that the TSS concentration decreased with distance from the shore, while the Chla concentration showed an initially decreasing trend followed by an increasing trend; moreover, these two water quality parameters presented different inter-annual variations. Then, we discussed the correlation between the changes in the TSS and Chla concentrations and the WS and PR variables. The contribution of this research is reflected in two aspects: 1. variations in water quality parameters over a wide range of water bodies can be evaluated based on MODIS data; 2. data from different time periods showed that the fluctuations of meteorological elements had different impacts on water bodies based on the distance from the shore. The results provide new insights for the management of estuary water environments

A Hybrid Framework for Simulating Actual Evapotranspiration in Data-Deficient Areas: A Case Study of the Inner Mongolia Section of the Yellow River Basin

Evapotranspiration (ET) plays an important role in transferring water and converting energy in the land–atmosphere system. Accurately estimating ET is crucial for understanding global climate change, ecological environmental problems, the water cycle, and hydrological processes. Machine learning (ML) algorithms have been considered as a promising method for estimating ET in recent years. However, due to the limitations associated with the spatial–temporal resolution of the flux tower data commonly used as the target set in ML algorithms, the ability of ML to discover the inherent laws within the data is reduced. In this study, a hybrid framework was established to simulate ET in data-deficient areas. ET simulation results of a coupled model comprising the Budyko function and complementary principle (BC2021) were used as the target set of the random forest model, instead of using the flux station observation data. By combining meteorological and hydrological data, the monthly ET of the Inner Mongolia section of the Yellow River Basin (IMSYRB) was simulated from 1982 to 2020, and good results were obtained (R2 = 0.94, MAE = 3.82 mm/mon, RMSE = 5.07 mm/mon). Furthermore, the temporal and spatial variations in ET and the influencing factors were analysed. In the past 40 years, annual ET in the IMSYRB ranged between 241.38 mm and 326.37 mm, showing a fluctuating growth trend (slope = 0.80 mm/yr), and the summer ET accounted for the highest proportion in the year. Spatially, ET in the IMSYRB showed a regular distribution of high ET in the eastern region and low ET in the western area. The high ET value areas gradually expanded from east to west over time, and the area increased continuously, with the largest increase observed in the 1980s. Temperature, precipitation, and normalized difference vegetation index (NDVI) were found to be the most important factors affecting ET in the region and play a positive role in promoting ET changes. These results provide an excellent example of long-term and large-scale accurate ET simulations in an area with sparse flux stations

Transport Properties of Electro-Sprayed Polytetrafluoroethylene Fibrous Layer Filled with Aerogels/Phase Change Materials

This work is the first attempt to prepare microporous polytetrafluoroethylene (PTFE) fibrous layers embedded with aerogels/phase change materials. For preparation of this layer, the needle-less electrospray technology of water dispersion of individual components is used. Microstructure characteristics, including surface morphology and particle size distribution, and various properties of the prepared materials were investigated and explained. Transport performance of the fibrous layers embedded with aerogels/phase change materials, such as the transmission of heat, air, and water vapor was evaluated and discussed in details. It was found that the electro-sprayed materials composed by spherical particles with rough surface had compact disordered stacking structure. Aerogels and phase change materials (PCMs) play different roles in determining structural parameters and transport properties of the materials. Those parameters and properties could be flexibly adjusted by optimizing the spinning parameters, changing the content or proportion of the fillers to meet specific requirements