338 research outputs found
Penalty Decoding: Well Suppress the Self-Reinforcement Effect in Open-Ended Text Generation
The decoding algorithm is critical for open-ended text generation,
transforming latent representations into coherent and meaningful outputs. This
paper investigates the self-reinforcement effect in text generation and the
effectiveness of a repetition penalty to mitigate it. However, determining the
optimal repetition penalty value is challenging. To tackle this, we propose a
forgetting mechanism that disregards distant tokens, reducing the burden of
penalty selection. In addition, we introduce a length penalty to address overly
short sentences caused by excessive penalties. Our penalty decoding approach
incorporating three strategies helps resolve issues with sampling methods
deviating from factual information. Experimental results demonstrate the
efficacy of our approach in generating high-quality sentences resembling human
output.Comment: Accepted by EMNLP202
Enable Language Models to Implicitly Learn Self-Improvement From Data
Large Language Models (LLMs) have demonstrated remarkable capabilities in
open-ended text generation tasks. However, the inherent open-ended nature of
these tasks implies that there is always room for improvement in the quality of
model responses. To address this challenge, various approaches have been
proposed to enhance the performance of LLMs. There has been a growing focus on
enabling LLMs to self-improve their response quality, thereby reducing the
reliance on extensive human annotation efforts for collecting diverse and
high-quality training data. Recently, prompting-based methods have been widely
explored among self-improvement methods owing to their effectiveness,
efficiency, and convenience. However, those methods usually require explicitly
and thoroughly written rubrics as inputs to LLMs. It is expensive and
challenging to manually derive and provide all necessary rubrics with a
real-world complex goal for improvement (e.g., being more helpful and less
harmful). To this end, we propose an ImPlicit Self-ImprovemenT (PIT) framework
that implicitly learns the improvement goal from human preference data. PIT
only requires preference data that are used to train reward models without
extra human efforts. Specifically, we reformulate the training objective of
reinforcement learning from human feedback (RLHF) -- instead of maximizing
response quality for a given input, we maximize the quality gap of the response
conditioned on a reference response. In this way, PIT is implicitly trained
with the improvement goal of better aligning with human preferences.
Experiments on two real-world datasets and one synthetic dataset show that our
method significantly outperforms prompting-based methods.Comment: 28 pages, 5 figures, 4 table
Laboratory Study on Improving Recovery of Ultra-Heavy Oil Using High-Temperature-Resistant Foam
After multiple rounds of steam huff-and-puff processes, an ultra-heavy oil reservoir is prone to excessive steam injection pressure, large heat loss, small sweep range of steam, and steam channeling, thus severely affecting the effective utilization of the oil reservoir. To solve these problems, one-dimensional and three-dimensional (3D) physical simulation tools were used to study the plugging performance of high-temperature composite foams by adding tanning extract and alkali lignin under the influence of some factors such as the reservoir temperature, salinity of formation water, and injection methods. The ultra-heavy oil used in the experiment comes from Shengli Oilfield. Under the condition of surface degassing, the viscosity of ultra-heavy oil could reach 145169 mPa.s at 60 °C. The experimental results show that the foam can produce a synergistic effect with both gel systems, indicating that the gel increases the stability of the foam. The foam can transfer more gel into the high-permeability formation, which can efficiently control the foam. The 3D physical simulation experiments indicated that both the systems enhance the recovery of heavy oil reservoir and reduce its moisture content significantly using steam injection. The method involving tannin extract foam and steam injection increased the recovery by 20% compared to the foam involving only steam injection. The method involving alkali lignin foam and steam injection increased the recovery by 11%
Identification and characterization of insect-specific proteins by genome data analysis
Background: Insects constitute the vast majority of known species with their importance including biodiversity, agricultural, and human health concerns. It is likely that the successful adaptation of the Insecta clade depends on specific components in its proteome that give rise to specialized features. However, proteome determination is an intensive undertaking. Here we present results from a computational method that uses genome analysis to characterize insect and eukaryote proteomes as an approximation complementary to experimental approaches.
Results: Homologs in common to Drosophila melanogaster, Anopheles gambiae, Bombyx mori, Tribolium castaneum, and Apis mellifera were compared to the complete genomes of three non-insect eukaryotes (opisthokonts) Homo sapiens, Caenorhabditis elegans and Saccharomyces cerevisiae. This operation yielded 154 groups of orthologous proteins in Drosophila to be insect-specific homologs; 466 groups were determined to be common to eukaryotes (represented by three opisthokonts). ESTs from the hemimetabolous insect Locust migratoria were also considered in order to approximate their corresponding genes in the insect-specific homologs. Stress and stimulus response proteins were found to constitute a higher fraction in the insect-specific homologs than in the homologs common to eukaryotes.
Conclusion: The significant representation of stress response and stimulus response proteins in proteins determined to be insect-specific, along with specific cuticle and pheromone/odorant binding proteins, suggest that communication and adaptation to environments may distinguish insect evolution relative to other eukaryotes. The tendency for low Ka/Ks ratios in the insect-specific protein set suggests purifying selection pressure. The generally larger number of paralogs in the insect-specific proteins may indicate adaptation to environment changes. Instances in our insect-specific protein set have been arrived at through experiments reported in the literature, supporting the accuracy of our approach
FGF: A web tool for Fishing Gene Family in a whole genome database
Gene duplication is an important process in evolution. The availability of genome sequences of a number of organisms has made it possible to conduct comprehensive searches for duplicated genes enabling informative studies of their evolution. We have established the FGF (Fishing Gene Family) program to efficiently search for and identify gene families. The FGF output displays the results as visual phylogenetic trees including information on gene structure, chromosome position, duplication fate and selective pressure. It is particularly useful to identify pseudogenes and detect changes in gene structure. FGF is freely available on a web server at http://fgf.genomics.org.cn
Augmentation with Projection: Towards an Effective and Efficient Data Augmentation Paradigm for Distillation
Knowledge distillation is one of the primary methods of transferring
knowledge from large to small models. However, it requires massive
task-specific data, which may not be plausible in many real-world applications.
Data augmentation methods such as representation interpolation, token
replacement, or augmentation with models are applied to tackle this problem.
However, these data augmentation methods either potentially cause shifts in
decision boundaries (representation interpolation), are not expressive enough
(token replacement), or introduce too much computational overhead (augmentation
with models). To this end, we propose AugPro (Augmentation with Projection), an
effective and efficient data augmentation method for distillation. Our method
builds on top of representation interpolation augmentation methods to maintain
the diversity of expressions and converts the augmented data to tokens to avoid
shifting decision boundaries. It uses simple operations that come with little
computational overhead. The results on multiple GLUE tasks show that our
methods can improve distillation performance by a large margin at a low time
cost. Codes are available at
https://github.com/google-research/google-research/tree/master/augpro.Comment: 20 pages, 5 figures. Accepted by ICLR 202
Prevalence and predictor for malignancy of contralateral thyroid nodules in patients with unilateral PTMC: a systematic review and meta-analysis
Background: The presence of clinically negative nodules on the contralater al lobe is common in patients with unilateral papillary thyroid microcarci noma (PTMC). The appropriate operational strategies of contralateral thyroid nodules remain controversial. In this study, we analyzed clinical features that could be pred ictors for malignancy of contralateral thyroid nodules coexisting with diagnosed unilateral PTMC.
Methods: The literatures published from January 2000 to December 2019 w ere searched in PubMed, Cochrane Library, Embase, Web of Science, CNKI, and Wan Fang database. Odds ratio (OR) with 95% CI was used to describe categorical va riables. Heterogeneity among studies was examined by the Q test and I 2 test; potential publication bias was detected by Harbord test and ‘trim and fill’ method.
Results: In this meta-analysis, 2541 studies were searched and 8 studie s were finally included. The results showed that the rate of carcinoma in cont ralateral nodules was 23% (OR = 0.23, 95% CI = 0.18–0.29). The pooled data indicated that contralateral malignancy was not associated with age, gender, primary lesion size, ipsilateral central lymph node metastasis and multifocality of contralateral lesion . The following variables have correlations with an increased risk of contralateral malig nancy: multifocality of primary carcinomas (OR = 3.93, 95% CI = 2.70–5.73, P < 0.0001), capsular invasion (OR = 1.61, 95% CI = 1.10–2.36, P = 0.01), and Hashimoto's thyroiditis (OR = 1.57, 95% CI = 1.13–2.20, P = 0.008).
Conclusions: Based on our meta-analysis, the rate at which contralateral ma lignancies are preoperatively misdiagnosed as benign is 23%. The risk fact ors for contralateral malignancy in unilateral PTMC patients with contralateral clini cal negative nodules include multifocality of primary carcinomas, capsular invasion, and Has himoto's thyroiditis
CLEAN-EVAL: Clean Evaluation on Contaminated Large Language Models
We are currently in an era of fierce competition among various large language
models (LLMs) continuously pushing the boundaries of benchmark performance.
However, genuinely assessing the capabilities of these LLMs has become a
challenging and critical issue due to potential data contamination, and it
wastes dozens of time and effort for researchers and engineers to download and
try those contaminated models. To save our precious time, we propose a novel
and useful method, Clean-Eval, which mitigates the issue of data contamination
and evaluates the LLMs in a cleaner manner. Clean-Eval employs an LLM to
paraphrase and back-translate the contaminated data into a candidate set,
generating expressions with the same meaning but in different surface forms. A
semantic detector is then used to filter the generated low-quality samples to
narrow down this candidate set. The best candidate is finally selected from
this set based on the BLEURT score. According to human assessment, this best
candidate is semantically similar to the original contamination data but
expressed differently. All candidates can form a new benchmark to evaluate the
model. Our experiments illustrate that Clean-Eval substantially restores the
actual evaluation results on contaminated LLMs under both few-shot learning and
fine-tuning scenarios
- …