47 research outputs found
On an adaptive preconditioned Crank-Nicolson MCMC algorithm for infinite dimensional Bayesian inferences
Many scientific and engineering problems require to perform Bayesian
inferences for unknowns of infinite dimension. In such problems, many standard
Markov Chain Monte Carlo (MCMC) algorithms become arbitrary slow under the mesh
refinement, which is referred to as being dimension dependent. To this end, a
family of dimensional independent MCMC algorithms, known as the preconditioned
Crank-Nicolson (pCN) methods, were proposed to sample the infinite dimensional
parameters. In this work we develop an adaptive version of the pCN algorithm,
where the covariance operator of the proposal distribution is adjusted based on
sampling history to improve the simulation efficiency. We show that the
proposed algorithm satisfies an important ergodicity condition under some mild
assumptions. Finally we provide numerical examples to demonstrate the
performance of the proposed method
A hybrid adaptive MCMC algorithm in function spaces
The preconditioned Crank-Nicolson (pCN) method is a Markov Chain Monte Carlo
(MCMC) scheme, specifically designed to perform Bayesian inferences in function
spaces. Unlike many standard MCMC algorithms, the pCN method can preserve the
sampling efficiency under the mesh refinement, a property referred to as being
dimension independent. In this work we consider an adaptive strategy to further
improve the efficiency of pCN. In particular we develop a hybrid adaptive MCMC
method: the algorithm performs an adaptive Metropolis scheme in a chosen finite
dimensional subspace, and a standard pCN algorithm in the complement space of
the chosen subspace. We show that the proposed algorithm satisfies certain
important ergodicity conditions. Finally with numerical examples we demonstrate
that the proposed method has competitive performance with existing adaptive
algorithms.Comment: arXiv admin note: text overlap with arXiv:1511.0583
ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats
In the complex domain of large language models (LLMs), striking a balance
between computational efficiency and maintaining model quality is a formidable
challenge. Navigating the inherent limitations of uniform quantization,
particularly when dealing with outliers, and motivated by the launch of
NVIDIA's H100 hardware, this study delves into the viability of floating-point
(FP) quantization, particularly focusing on FP8 and FP4, as a potential
solution. Our comprehensive investigation reveals that for LLMs, FP8 activation
consistently outshines its integer (INT8) equivalent, with the performance edge
becoming more noticeable in models possessing parameters beyond one billion.
For weight quantization, our findings indicate that FP4 exhibits comparable, if
not superior, performance to INT4, simplifying deployment on FP-supported
hardware like H100. To mitigate the overhead from precision alignment caused by
the disparity between weights and activations, we propose two scaling
constraints for weight quantization that negligibly impact the performance
compared to the standard W4A8 model. We additionally enhance our quantization
methods by integrating the Low Rank Compensation (LoRC) strategy, yielding
improvements especially in smaller models. The results of our investigation
emphasize the immense potential of FP quantization for LLMs, paving the way for
high-efficiency deployment in resource-limited settings
Inefficiency of K-FAC for Large Batch Size Training
In stochastic optimization, using large batch sizes during training can
leverage parallel resources to produce faster wall-clock training times per
training epoch. However, for both training loss and testing error, recent
results analyzing large batch Stochastic Gradient Descent (SGD) have found
sharp diminishing returns, beyond a certain critical batch size. In the hopes
of addressing this, it has been suggested that the Kronecker-Factored
Approximate Curvature (\mbox{K-FAC}) method allows for greater scalability to
large batch sizes, for non-convex machine learning problems such as neural
network optimization, as well as greater robustness to variation in model
hyperparameters. Here, we perform a detailed empirical analysis of large batch
size training %of these two hypotheses, for both \mbox{K-FAC} and SGD,
evaluating performance in terms of both wall-clock time and aggregate
computational cost. Our main results are twofold: first, we find that both
\mbox{K-FAC} and SGD doesn't have ideal scalability behavior beyond a certain
batch size, and that \mbox{K-FAC} does not exhibit improved large-batch
scalability behavior, as compared to SGD; and second, we find that
\mbox{K-FAC}, in addition to requiring more hyperparameters to tune, suffers
from similar hyperparameter sensitivity behavior as does SGD. We discuss
extensive results using ResNet and AlexNet on \mbox{CIFAR-10} and SVHN,
respectively, as well as more general implications of our findings