14 research outputs found
QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models
Mixture-of-Experts (MoE) architectures offer a general solution to the high
inference costs of large language models (LLMs) via sparse routing, bringing
faster and more accurate models, at the cost of massive parameter counts. For
example, the SwitchTransformer-c2048 model has 1.6 trillion parameters,
requiring 3.2TB of accelerator memory to run efficiently, which makes practical
deployment challenging and expensive. In this paper, we present a solution to
this memory problem, in form of a new compression and execution framework
called QMoE. Specifically, QMoE consists of a scalable algorithm which
accurately compresses trillion-parameter MoEs to less than 1 bit per parameter,
in a custom format co-designed with bespoke GPU decoding kernels to facilitate
efficient end-to-end compressed inference, with minor runtime overheads
relative to uncompressed execution. Concretely, QMoE can compress the 1.6
trillion parameter SwitchTransformer-c2048 model to less than 160GB (20x
compression, 0.8 bits per parameter) at only minor accuracy loss, in less than
a day on a single GPU. This enables, for the first time, the execution of a
trillion-parameter model on affordable commodity hardware, like a single server
with 4x NVIDIA A6000 or 8x NVIDIA 3090 GPUs, at less than 5% runtime overhead
relative to ideal uncompressed inference. The source code and compressed models
are available at github.com/IST-DASLab/qmoe
ZipLM: Inference-Aware Structured Pruning of Language Models
The breakthrough performance of large language models (LLMs) comes with major
computational footprints and high deployment costs. In this paper, we progress
towards resolving this problem by proposing a novel structured compression
approach for LLMs, called ZipLM. ZipLM achieves state-of-the-art
accuracy-vs-speedup, while matching a set of desired target runtime speedups in
any given inference environment. Specifically, given a model, a dataset, an
inference environment, as well as a set of speedup targets, ZipLM iteratively
identifies and removes components with the worst loss-runtime trade-off. Unlike
prior methods that specialize in either the post-training/one-shot or the
gradual compression setting, and only for specific families of models such as
BERT (encoder) or GPT (decoder), ZipLM produces state-of-the-art compressed
models across all these settings. Furthermore, ZipLM achieves superior results
for a fraction of the computational cost relative to prior distillation and
pruning techniques, making it a cost-effective approach for generating an
entire family of smaller, faster, and highly accurate models, guaranteed to
meet the desired inference specifications. In particular, ZipLM outperforms all
prior BERT-base distillation and pruning techniques, such as CoFi, MiniLM, and
TinyBERT. Moreover, it matches the performance of the heavily optimized
MobileBERT model, obtained via extensive architecture search, by simply pruning
the baseline BERT-large model. When compressing GPT2, ZipLM outperforms
DistilGPT2 while being 60% smaller and 30% faster. Our code is available at:
https://github.com/IST-DASLab/ZipLM.Comment: Accepted to NeurIPS 202
L-GreCo: Layerwise-Adaptive Gradient Compression for Efficient and Accurate Deep Learning
Data-parallel distributed training of deep neural networks (DNN) has gained
very widespread adoption, but can still experience communication bottlenecks.
To address this issue, entire families of compression mechanisms have been
developed, including quantization, sparsification, and low-rank approximation,
some of which are seeing significant practical adoption. Despite this progress,
almost all known compression schemes apply compression uniformly across DNN
layers, although layers are heterogeneous in terms of parameter count and their
impact on model accuracy. In this work, we provide a general framework for
adapting the degree of compression across the model's layers dynamically during
training, improving the overall compression, while leading to substantial
speedups, without sacrificing accuracy. Our framework, called L-GreCo, is based
on an adaptive algorithm, which automatically picks the optimal compression
parameters for model layers guaranteeing the best compression ratio while
satisfying an error constraint. Extensive experiments over image classification
and language modeling tasks shows that L-GreCo is effective across all existing
families of compression methods, and achieves up to 2.5 training
speedup and up to 5 compression improvement over efficient
implementations of existing approaches, while recovering full accuracy.
Moreover, L-GreCo is complementary to existing adaptive algorithms, improving
their compression ratio by 50% and practical throughput by 66%
QIGen: Generating Efficient Kernels for Quantized Inference on Large Language Models
We present ongoing work on a new automatic code generation approach for
supporting quantized generative inference on LLMs such as LLaMA or OPT on
off-the-shelf CPUs. Our approach is informed by the target architecture and a
performance model, including both hardware characteristics and method-specific
accuracy constraints. Results on CPU-based inference for LLaMA models show that
our approach can lead to high performance and high accuracy, comparing
favorably to the best existing open-source solution. A preliminary
implementation is available at https://github.com/IST-DASLab/QIGen
On the Sample Complexity of Adversarial Multi-Source PAC Learning
We study the problem of learning from multiple untrusted data sources, a
scenario of increasing practical relevance given the recent emergence of
crowdsourcing and collaborative learning paradigms. Specifically, we analyze
the situation in which a learning system obtains datasets from multiple
sources, some of which might be biased or even adversarially perturbed. It is
known that in the single-source case, an adversary with the power to corrupt a
fixed fraction of the training data can prevent PAC-learnability, that is, even
in the limit of infinitely much training data, no learning system can approach
the optimal test error. In this work we show that, surprisingly, the same is
not true in the multi-source setting, where the adversary can arbitrarily
corrupt a fixed fraction of the data sources. Our main results are a
generalization bound that provides finite-sample guarantees for this learning
setting, as well as corresponding lower bounds. Besides establishing
PAC-learnability our results also show that in a cooperative learning setting
sharing data with other parties has provable benefits, even if some
participants are malicious.Comment: International Conference on Machine Learning (ICML) 2020:
Camera-ready. Strengthened the definition of adversarial PAC-learnability,
added explicit bounds on sample complexit
Sparse Fine-tuning for Inference Acceleration of Large Language Models
We consider the problem of accurate sparse fine-tuning of large language
models (LLMs), that is, fine-tuning pretrained LLMs on specialized tasks, while
inducing sparsity in their weights. On the accuracy side, we observe that
standard loss-based fine-tuning may fail to recover accuracy, especially at
high sparsities. To address this, we perform a detailed study of
distillation-type losses, determining an L2-based distillation approach we term
SquareHead which enables accurate recovery even at higher sparsities, across
all model types. On the practical efficiency side, we show that sparse LLMs can
be executed with speedups by taking advantage of sparsity, for both CPU and GPU
runtimes. While the standard approach is to leverage sparsity for computational
reduction, we observe that in the case of memory-bound LLMs sparsity can also
be leveraged for reducing memory bandwidth. We exhibit end-to-end results
showing speedups due to sparsity, while recovering accuracy, on T5 (language
translation), Whisper (speech translation), and open GPT-type (MPT for text
generation). For MPT text generation, we show for the first time that sparse
fine-tuning can reach 75% sparsity without accuracy drops, provide notable
end-to-end speedups for both CPU and GPU inference, and highlight that sparsity
is also compatible with quantization approaches. Models and software for
reproducing our results are provided in Section 6
On the sample complexity of adversarial multi-source PAC learning
We study the problem of learning from multiple untrusted data sources, a scenario of increasing practical relevance given the recent emergence of crowdsourcing and collaborative learning paradigms. Specifically, we analyze the situation in which a learning system obtains datasets from multiple sources, some of which might be biased or even adversarially perturbed. It is
known that in the single-source case, an adversary with the power to corrupt a fixed fraction of the training data can prevent PAC-learnability, that is, even in the limit of infinitely much training data, no learning system can approach the optimal test error. In this work we show that, surprisingly, the same is not true in the multi-source setting, where the adversary can arbitrarily
corrupt a fixed fraction of the data sources. Our main results are a generalization bound that provides finite-sample guarantees for this learning setting, as well as corresponding lower bounds. Besides establishing PAC-learnability our results also show that in a cooperative learning setting sharing data with other parties has provable benefits, even if some
participants are malicious
Accurate Neural Network Pruning Requires Rethinking Sparse Optimization
Obtaining versions of deep neural networks that are both highly-accurate and
highly-sparse is one of the main challenges in the area of model compression,
and several high-performance pruning techniques have been investigated by the
community. Yet, much less is known about the interaction between sparsity and
the standard stochastic optimization techniques used for training sparse
networks, and most existing work uses standard dense schedules and
hyperparameters for training sparse networks. In this work, we examine the
impact of high sparsity on model training using the standard computer vision
and natural language processing sparsity benchmarks. We begin by showing that
using standard dense training recipes for sparse training is suboptimal, and
results in under-training. We provide new approaches for mitigating this issue
for both sparse pre-training of vision models (e.g. ResNet50/ImageNet) and
sparse fine-tuning of language models (e.g. BERT/GLUE), achieving
state-of-the-art results in both settings in the high-sparsity regime, and
providing detailed analyses for the difficulty of sparse training in both
scenarios. Our work sets a new threshold in terms of the accuracies that can be
achieved under high sparsity, and should inspire further research into
improving sparse model training, to reach higher accuracies under high
sparsity, but also to do so efficiently
QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models
Large Language Models (LLMs) from the GPT family have become extremely
popular, leading to a race towards reducing their inference costs to allow for
efficient local computation. Yet, the vast majority of existing work focuses on
weight-only quantization, which can reduce runtime costs in the memory-bound
one-token-at-a-time generative setting, but does not address them in
compute-bound scenarios, such as batched inference or prompt processing. In
this paper, we address the general quantization problem, where both weights and
activations should be quantized. We show, for the first time, that the majority
of inference computations for large generative models such as LLaMA, OPT, and
Falcon can be performed with both weights and activations being cast to 4 bits,
in a way that leads to practical speedups, while at the same time maintaining
good accuracy. We achieve this via a hybrid quantization strategy called QUIK,
which compresses most of the weights and activations to 4-bit, while keeping
some outlier weights and activations in higher-precision. The key feature of
our scheme is that it is designed with computational efficiency in mind: we
provide GPU kernels matching the QUIK format with highly-efficient layer-wise
runtimes, which lead to practical end-to-end throughput improvements of up to
3.4x relative to FP16 execution. We provide detailed studies for models from
the OPT, LLaMA-2 and Falcon families, as well as a first instance of accurate
inference using quantization plus 2:4 sparsity. Code is available at:
https://github.com/IST-DASLab/QUIK.Comment: 16 page
JaxPruner: A concise library for sparsity research
This paper introduces JaxPruner, an open-source JAX-based pruning and sparse
training library for machine learning research. JaxPruner aims to accelerate
research on sparse neural networks by providing concise implementations of
popular pruning and sparse training algorithms with minimal memory and latency
overhead. Algorithms implemented in JaxPruner use a common API and work
seamlessly with the popular optimization library Optax, which, in turn, enables
easy integration with existing JAX based libraries. We demonstrate this ease of
integration by providing examples in four different codebases: Scenic, t5x,
Dopamine and FedJAX and provide baseline experiments on popular benchmarks.Comment: Jaxpruner is hosted at http://github.com/google-research/jaxprune