Suffix Retrieval-Augmented Language Modeling

Causal language modeling (LM) uses word history to predict the next word. BERT, on the other hand, makes use of bi-directional word information in a sentence to predict words at masked positions. While BERT is effective in sequence encoding, it is non-causal by nature and is not designed for sequence generation. In this paper, we propose a novel language model, SUffix REtrieval-Augmented LM (SUREALM), that simulates a bi-directional contextual effect in an autoregressive manner. SUREALM employs an embedding retriever to search for training sentences in a data store that share similar word history during sequence generation. In particular, the suffix portions of the retrieved sentences mimick the "future" context. We evaluated our proposed model on the DSTC9 spoken dialogue corpus and showed promising word perplexity reduction on the validation and test set compared to competitive baselines.Comment: 5 pages, 1 figure. Submitted to ICASSP 202

Beyond the photocycle-how cryptochromes regulate photoresponses in plants?

Cryptochromes (CRYs) are blue light receptors that mediate light regulation of plant growth and development. Land plants possess various numbers of cryptochromes, CRY1 and CRY2, which serve overlapping and partially redundant functions in different plant species. Cryptochromes exist as physiologically inactive monomers in darkness; photoexcited cryptochromes undergo homodimerization to increase their affinity to the CRY-signaling proteins, such as CIBs (CRY2-interacting bHLH), PIFs (Phytochrome-Interacting Factors), AUX/IAA (Auxin/INDOLE-3-ACETIC ACID), and the COP1-SPAs (Constitutive Photomorphogenesis 1-Suppressors of Phytochrome A) complexes. These light-dependent protein-protein interactions alter the activity of the CRY-signaling proteins to change gene expression and developmental programs in response to light. In the meantime, photoexcitation also changes the affinity of cryptochromes to the CRY-regulatory proteins, such as BICs (Blue-light Inhibitors of CRYs) and PPKs (Photoregulatory Protein Kinases), to modulate the activity, modification, or abundance of cryptochromes and photosensitivity of plants in response to the changing light environment

Pre-training with Synthetic Data Helps Offline Reinforcement Learning

Recently, it has been shown that for offline deep reinforcement learning (DRL), pre-training Decision Transformer with a large language corpus can improve downstream performance (Reid et al., 2022). A natural question to ask is whether this performance gain can only be achieved with language pre-training, or can be achieved with simpler pre-training schemes which do not involve language. In this paper, we first show that language is not essential for improved performance, and indeed pre-training with synthetic IID data for a small number of updates can match the performance gains from pre-training with a large language corpus; moreover, pre-training with data generated by a one-step Markov chain can further improve the performance. Inspired by these experimental results, we then consider pre-training Conservative Q-Learning (CQL), a popular offline DRL algorithm, which is Q-learning-based and typically employs a Multi-Layer Perceptron (MLP) backbone. Surprisingly, pre-training with simple synthetic data for a small number of updates can also improve CQL, providing consistent performance improvement on D4RL Gym locomotion datasets. The results of this paper not only illustrate the importance of pre-training for offline DRL but also show that the pre-training data can be synthetic and generated with remarkably simple mechanisms.Comment: 28 pages, 7 figure

Detoxify Language Model Step-by-Step

Detoxification for LLMs is challenging since it requires models to avoid generating harmful content while maintaining the generation capability. To ensure the safety of generations, previous detoxification methods detoxify the models by changing the data distributions or constraining the generations from different aspects in a single-step manner. However, these approaches will dramatically affect the generation quality of LLMs, e.g., discourse coherence and semantic consistency, since language models tend to generate along the toxic prompt while detoxification methods work in the opposite direction. To handle such a conflict, we decompose the detoxification process into different sub-steps, where the detoxification is concentrated in the input stage and the subsequent continual generation is based on the non-toxic prompt. Besides, we also calibrate the strong reasoning ability of LLMs by designing a Detox-Chain to connect the above sub-steps in an orderly manner, which allows LLMs to detoxify the text step-by-step. Automatic and human evaluation on two benchmarks reveals that by training with Detox-Chain, six LLMs scaling from 1B to 33B can obtain significant detoxification and generation improvement. Our code and data are available at https://github.com/CODINNLG/Detox-CoT. Warning: examples in the paper may contain uncensored offensive content

Calculation of Stability Limit Displacement of Surrounding Rock of Deep-Buried Soft Rock Tunnel Construction Based on Fuzzy Logic Matching Algorithm

With the continuous development of society and economy, infrastructure construction is expanding on a large scale. The stress concentration after excavation in deep-buried soft rock masses may cause the stress level of the rock mass to exceed the strength of the surrounding rock and form plastic stress, the area where plastic shear slip or plastic flow occurs. Based on the fuzzy logic matching algorithm for the surrounding rock of deep-buried soft rock tunnel construction, this paper analyzes the development law of surrounding rock deformation and supporting force over time in actual construction by establishing elementary function mathematical calculation equations, and tries to construct a set of It is used to determine the actual deep-buried soft rock tunnel surrounding rock stability limit displacement value process. Practical results show that the process based on fuzzy logic matching algorithm can effectively meet the requirements of actual deep-buried soft rock tunnel engineering

Nonlinear random extrapolation estimates of π\pi under Dirichlet distributions

We construct optimal nonlinear extrapolation estimates of $\pi$ based on random cyclic polygons generated from symmetric Dirichlet distributions. While the semiperimeter $S_n$ and the area $A_n$ of such random inscribed polygons and the semiperimeter (and area) $S_n'$ of the corresponding random circumscribing polygons are known to converge to $\pi$ w.p.$1$ and their distributions are also asymptotically normal as $n \to \infty$, we study in this paper nonlinear extrapolations of the forms $\mathcal{W}_n = S_n^{\alpha} A_n^{\beta} S_n'^{\, \gamma}$ and $\mathcal{W}_n (p) = ( \alpha S_n^p + \beta A_n^p + \gamma S_n'^{\, p} )^{1/p}$ where $\alpha + \beta + \gamma = 1$ and $p \neq 0$. By deriving probabilistic asymptotic expansions with carefully controlled error estimates, we show that $\mathcal{W}_n$ and $\mathcal{W}_n (p)$ also converge to $\pi$ w.p.$1$ and are asymptotically normal. Furthermore, to minimize the approximation error associated with $\mathcal{W}_n$ and $\mathcal{W}_n (p)$, the parameters must satisfy the optimality condition $\alpha + 4 \beta - 2 \gamma = 0$. Our results generalize previous work on nonlinear extrapolations of $\pi$ which employ inscribed polygons only and the vertices are also assumed to be independently and uniformly distributed on the unit circle

Robust Unstructured Knowledge Access in Conversational Dialogue with ASR Errors

Performance of spoken language understanding (SLU) can be degraded with automatic speech recognition (ASR) errors. We propose a novel approach to improve SLU robustness by randomly corrupting clean training text with an ASR error simulator, followed by self-correcting the errors and minimizing the target classification loss in a joint manner. In the proposed error simulator, we leverage confusion networks generated from an ASR decoder without human transcriptions to generate a variety of error patterns for model training. We evaluate our approach on the DSTC10 challenge targeted for knowledge-grounded task-oriented conversational dialogues with ASR errors. Experimental results show the effectiveness of our proposed approach, boosting the knowledge-seeking turn detection (KTD) F1 significantly from 0.9433 to 0.9904. Knowledge cluster classification is boosted from 0.7924 to 0.9333 in Recall@1. After knowledge document re-ranking, our approach shows significant improvement in all knowledge selection metrics, from 0.7358 to 0.7806 in Recall@1, from 0.8301 to 0.9333 in Recall@5, and from 0.7798 to 0.8460 in MRR@5 on the test set. In the recent DSTC10 evaluation, our approach demonstrates significant improvement in knowledge selection, boosting Recall@1 from 0.495 to 0.7144 compared to the official baseline. Our source code is released in GitHub https://github.com/yctam/dstc10_track2_task2.git.Comment: 7 pages, 2 figures. Accepted at ICASSP 202

A CRY-BIC negative-feedback circuitry regulating blue light sensitivity of Arabidopsis.

Cryptochromes are blue light receptors that regulate various light responses in plants. Arabidopsis cryptochrome 1 (CRY1) and cryptochrome 2 (CRY2) mediate blue light inhibition of hypocotyl elongation and long-day (LD) promotion of floral initiation. It has been reported recently that two negative regulators of Arabidopsis cryptochromes, Blue light Inhibitors of Cryptochromes 1 and 2 (BIC1 and BIC2), inhibit cryptochrome function by blocking blue light-dependent cryptochrome dimerization. However, it remained unclear how cryptochromes regulate the BIC gene activity. Here we show that cryptochromes mediate light activation of transcription of the BIC genes, by suppressing the activity of CONSTITUTIVE PHOTOMORPHOGENIC 1 (COP1), resulting in activation of the transcription activator ELONGATED HYPOCOTYL 5 (HY5) that is associated with chromatins of the BIC promoters. These results demonstrate a CRY-BIC negative-feedback circuitry that regulates the activity of each other. Surprisingly, phytochromes also mediate light activation of BIC transcription, suggesting a novel photoreceptor co-action mechanism to sustain blue light sensitivity of plants under the broad spectra of solar radiation in nature

Formation and Distribution of Tight Sand Gas Reservoirs in the Sichuan Basin, China

Located in the central west of China, the Sichuan Basin is abundant in natural gas resources. It is the earliest basin where natural gas was discovered and utilized in the world. After more than 60 years’ exploration, many gas fields have been found in the Paleozoic - Mesozoic carbonate and clastic formations. The basin has become a critical production base in China for its cumulative proved recoverable gas reserves of more than 8000×108 m3, and gas production over 150×108 m3 in 2010. During the past decade, many tight gas reservoirs have been found in the Xujiahe coal measures of the upper Triassic continental deposits. Based on lithology, this suite of formation can be divided into six members from bottom to top. The source rocks are the coal beds and carbon-bearing mudstones in Xu1, Xu3 and Xu5 members with relatively high organic carbon contents and type III kerogen; the reservoir rocks are the tight sandstones in Xu2, Xu4 and Xu6 members. The source rocks and the reservoirs distribute alternatively and widely in “sandwiched” structure, providing favorable conditions for natural gas accumulating near source. As the formations are gentle and lack of structural traps, the lithologic gas reservoirs dominate the Xujiahe tight sandstones. Both coal-measure source rocks and sandstone reservoir distribute in strong heterogeneity, leading to thin gas-layers in the reservoir, poor continuity in plane, and varying full-up ratio and gas saturation in the gas reservoir. Within the 80,000 km2 area, the Xujiahe Formation has the features of widespread gas-bearing beds and local gas enrichment. The current high-yield gas wells are mainly distributed in the tectonic highs or fractured zones in the areas with effective source-reservoir assemblages. The resources assessment is made considering the tight gas accumulating intensively into reservoir. It reveals the favorable gas-bearing area up to 6-7×104 km2 and the estimated recoverable gas reserves of 2-3×1012 m3 in the Xujiahe Formation.Key words: Sichuan Basin; Tight sandstone; Coal measures; Tight gas; “Sandwiched” structure; Resource