6 research outputs found
Efficient Benchmarking (of Language Models)
The increasing versatility of language models LMs has given rise to a new
class of benchmarks that comprehensively assess a broad range of capabilities.
Such benchmarks are associated with massive computational costs reaching
thousands of GPU hours per model. However the efficiency aspect of these
evaluation efforts had raised little discussion in the literature. In this work
we present the problem of Efficient Benchmarking namely intelligently reducing
the computation costs of LM evaluation without compromising reliability. Using
the HELM benchmark as a test case we investigate how different benchmark design
choices affect the computation-reliability tradeoff. We propose to evaluate the
reliability of such decisions by using a new measure Decision Impact on
Reliability DIoR for short. We find for example that the current leader on HELM
may change by merely removing a low-ranked model from the benchmark and observe
that a handful of examples suffice to obtain the correct benchmark ranking.
Conversely a slightly different choice of HELM scenarios varies ranking widely.
Based on our findings we outline a set of concrete recommendations for more
efficient benchmark design and utilization practices leading to dramatic cost
savings with minimal loss of benchmark reliability often reducing computation
by x100 or more
Quantum Detection with Unknown States
We address the problem of distinguishing among a finite collection of quantum
states, when the states are not entirely known. For completely specified
states, necessary and sufficient conditions on a quantum measurement minimizing
the probability of a detection error have been derived. In this work, we assume
that each of the states in our collection is a mixture of a known state and an
unknown state. We investigate two criteria for optimality. The first is
minimization of the worst-case probability of a detection error. For the second
we assume a probability distribution on the unknown states, and minimize of the
expected probability of a detection error.
We find that under both criteria, the optimal detectors are equivalent to the
optimal detectors of an ``effective ensemble''. In the worst-case, the
effective ensemble is comprised of the known states with altered prior
probabilities, and in the average case it is made up of altered states with the
original prior probabilities.Comment: Refereed version. Improved numerical examples and figures. A few
typos fixe