3 research outputs found
Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models
We present Step-Back Prompting, a simple prompting technique that enables
LLMs to do abstractions to derive high-level concepts and first principles from
instances containing specific details. Using the concepts and principles to
guide the reasoning steps, LLMs significantly improve their abilities in
following a correct reasoning path towards the solution. We conduct experiments
of Step-Back Prompting with PaLM-2L models and observe substantial performance
gains on a wide range of challenging reasoning-intensive tasks including STEM,
Knowledge QA, and Multi-Hop Reasoning. For instance, Step-Back Prompting
improves PaLM-2L performance on MMLU Physics and Chemistry by 7% and 11%,
TimeQA by 27%, and MuSiQue by 7%
Unifying Language Learning Paradigms
Existing pre-trained models are generally geared towards a particular class
of problems. To date, there seems to be still no consensus on what the right
architecture and pre-training setup should be. This paper presents a unified
framework for pre-training models that are universally effective across
datasets and setups. We begin by disentangling architectural archetypes with
pre-training objectives -- two concepts that are commonly conflated. Next, we
present a generalized and unified perspective for self-supervision in NLP and
show how different pre-training objectives can be cast as one another and how
interpolating between different objectives can be effective. We then propose
Mixture-of-Denoisers (MoD), a pre-training objective that combines diverse
pre-training paradigms together. We furthermore introduce a notion of mode
switching, wherein downstream fine-tuning is associated with specific
pre-training schemes. We conduct extensive ablative experiments to compare
multiple pre-training objectives and find that our method pushes the
Pareto-frontier by outperforming T5 and/or GPT-like models across multiple
diverse setups. Finally, by scaling our model up to 20B parameters, we achieve
SOTA performance on 50 well-established supervised NLP tasks ranging from
language generation (with automated and human evaluation), language
understanding, text classification, question answering, commonsense reasoning,
long text reasoning, structured knowledge grounding and information retrieval.
Our model also achieve strong results at in-context learning, outperforming
175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on
one-shot summarization. We release Flax-based T5X model checkpoints for the 20B
model at
\url{https://github.com/google-research/google-research/tree/master/ul2}
HyperPrompt: Prompt-based Task-Conditioning of Transformers
Prompt-Tuning is a new paradigm for finetuning pre-trained language models in
a parameter-efficient way. Here, we explore the use of HyperNetworks to
generate hyper-prompts: we propose HyperPrompt, a novel architecture for
prompt-based task-conditioning of self-attention in Transformers. The
hyper-prompts are end-to-end learnable via generation by a HyperNetwork.
HyperPrompt allows the network to learn task-specific feature maps where the
hyper-prompts serve as task global memories for the queries to attend to, at
the same time enabling flexible information sharing among tasks. We show that
HyperPrompt is competitive against strong multi-task learning baselines with as
few as of additional task-conditioning parameters, achieving great
parameter and computational efficiency. Through extensive empirical
experiments, we demonstrate that HyperPrompt can achieve superior performances
over strong T5 multi-task learning baselines and parameter-efficient adapter
variants including Prompt-Tuning and HyperFormer++ on Natural Language
Understanding benchmarks of GLUE and SuperGLUE across many model sizes.Comment: Accepted to ICML 202