35 research outputs found
Efficient Data Learning for Open Information Extraction with Pre-trained Language Models
Open Information Extraction (OpenIE) is a fundamental yet challenging task in
Natural Language Processing, which involves extracting all triples (subject,
predicate, object) from a given sentence. While labeling-based methods have
their merits, generation-based techniques offer unique advantages, such as the
ability to generate tokens not present in the original sentence. However, these
generation-based methods often require a significant amount of training data to
learn the task form of OpenIE and substantial training time to overcome slow
model convergence due to the order penalty. In this paper, we introduce a novel
framework, OK-IE, that ingeniously transforms the task form of OpenIE into the
pre-training task form of the T5 model, thereby reducing the need for extensive
training data. Furthermore, we introduce an innovative concept of Anchor to
control the sequence of model outputs, effectively eliminating the impact of
order penalty on model convergence and significantly reducing training time.
Experimental results indicate that, compared to previous SOTA methods, OK-IE
requires only 1/100 of the training data (900 instances) and 1/120 of the
training time (3 minutes) to achieve comparable results
ExpNote: Black-box Large Language Models are Better Task Solvers with Experience Notebook
Black-box Large Language Models (LLMs) have shown great power in solving
various tasks and are considered general problem solvers. However, LLMs still
fail in many specific tasks although understand the task instruction. In this
paper, we focus on the problem of boosting the ability of black-box LLMs to
solve downstream tasks. We propose ExpNote, an automated framework to help LLMs
better adapt to unfamiliar tasks through reflecting and noting experiences from
training data and retrieving them from external memory during testing. We
evaluate ExpNote on multiple tasks and the experimental results demonstrate
that the proposed method significantly improves the performance of black-box
LLMs. The data and code are available at
https://github.com/forangel2014/ExpNoteComment: EMNLP 2023 finding
S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models
The rapid development of Large Language Models (LLMs) has led to great
strides in model capabilities like reasoning and long-context understanding.
However, as LLMs are able to process longer contexts, it becomes more
challenging to evaluate whether they have acquired certain capabilities, since
the length of text (e.g., 100K tokens) they can process far exceeds what humans
can reliably assess in a reasonable duration. In this paper, we propose using
complex synthetic tasks as a proxy evaluation method, and present S3Eval, a
Synthetic, Scalable, Systematic evaluation suite for LLMs evaluation. As a
synthetic benchmark, S3Eval enables the creation of any number of evaluation
examples that are theoretically invisible to LLMs, mitigating the test set
contamination issue. The synthetic nature of S3Eval provides users full control
over the dataset, allowing them to systematically probe LLM capabilities by
scaling text length and varying task difficulty across diverse scenarios. The
strong correlation between S3Eval performance and scores of real-world
benchmarks like Big-Bench Hard (BBH) demonstrates the soundness of using S3Eval
for evaluation of LLMs. The in-depth analysis also uncover additional insights,
including performance drop when the answer is sparsely distributed or located
in the middle context, as well as some counter-intuitive trends of model
performance.Comment: Work in progres
LMTuner: An user-friendly and highly-integrable Training Framework for fine-tuning Large Language Models
With the burgeoning development in the realm of large language models (LLMs),
the demand for efficient incremental training tailored to specific industries
and domains continues to increase. Currently, the predominantly employed
frameworks lack modular design, it often takes a lot of coding work to
kickstart the training of LLM. To address this, we present "LMTuner", a highly
usable, integrable, and scalable system for training LLMs expeditiously and
with minimal user-input. LMTuner comprises three main modules - the
Interaction, Training, and Inference Modules. We advocate that LMTuner's
usability and integrality alleviate the complexities in training large language
models. Remarkably, even a novice user could commence training large language
models within five minutes. Furthermore, it integrates DeepSpeed frameworks and
supports Efficient Fine-Tuning methodologies like Low Rank Adaptation (LoRA),
Quantized LoRA (QLoRA), etc., enabling the training of language models scaling
from 300M to a whopping 130B parameters using a single server. The LMTuner's
homepage (https://wengsyx.github.io/LMTuner/)and screencast video
(https://youtu.be/nsXmWOmN3rE) are now publicly available
Arctic Ocean Simulations in the CMIP6 Ocean Model Intercomparison Project (OMIP)
oai:publications.copernicus.org:gmdd107357Arctic Ocean simulations in 19 global ocean-sea ice models participating in the Ocean Model Intercomparison Project (OMIP) of the CMIP6 are evaluated in this paper. Our results indicate that no significant improvements were achieved in the Arctic Ocean simulations from the previous Coordinated Ocean-ice Reference Experiments phase II (CORE-II) to the current OMIP. Large model biases and inter-model spread exist in the simulated mean state of the halocline and Atlantic Water layer in the OMIP models. Most of the OMIP models suffer from too thick and deep Atlantic Water layer, too deep halocline base, and large fresh biases in the halocline. The OMIP models largely agree on the inter-annual and decadal variability of the Arctic Ocean freshwater content and volume/heat/freshwater transports through the Arctic Ocean gateways. The models can reproduce observed changes in volume, heat and freshwater transports through the gateways except for the Bering Strait. Overall, the performance of the Arctic Ocean simulations is similar between the CORE2-forced OMIP-1 and JRA55-do-forced OMIP-2.</p
Bipartite Graph Pre-training for Unsupervised Extractive Summarization with Graph Convolutional Auto-Encoders
Pre-trained sentence representations are crucial for identifying significant
sentences in unsupervised document extractive summarization. However, the
traditional two-step paradigm of pre-training and sentence-ranking, creates a
gap due to differing optimization objectives. To address this issue, we argue
that utilizing pre-trained embeddings derived from a process specifically
designed to optimize cohensive and distinctive sentence representations helps
rank significant sentences. To do so, we propose a novel graph pre-training
auto-encoder to obtain sentence embeddings by explicitly modelling
intra-sentential distinctive features and inter-sentential cohesive features
through sentence-word bipartite graphs. These pre-trained sentence
representations are then utilized in a graph-based ranking algorithm for
unsupervised summarization. Our method produces predominant performance for
unsupervised summarization frameworks by providing summary-worthy sentence
representations. It surpasses heavy BERT- or RoBERTa-based sentence
representations in downstream tasks.Comment: Accepted by the 2023 Conference on Empirical Methods in Natural
Language Processing (EMNLP 2023