2 research outputs found
Relative error streaming quantiles
Approximating ranks, quantiles, and distributions over streaming data is a central task in data analysis and monitoring. Given a stream of n items from a data universe U equipped with a total order, the task is to compute a sketch (data structure) of size poly (log(n), 1/ε). Given the sketch and a query item y ∈ U, one should be able to approximate its rank in the stream, i.e., the number of stream elements smaller than or equal to y. Most works to date focused on additive ε n error approximation, culminating in the KLL sketch that achieved optimal asymptotic behavior. This paper investigates multiplicative (1±ε)$-error approximations to the rank. Practical motivation for multiplicative error stems from demands to understand the tails of distributions, and hence for sketches to be more accurate near extreme values. The most space-efficient algorithms due to prior work store either O(log(ε2 n)/ε2) or O(log3(ε n)/ε) universe items. This paper presents a randomized algorithm storing O(log1.5 (ε n)/ε) items, which is within an O(√log(ε n)) factor of optimal. The algorithm does not require prior knowledge of the stream length and is fully mergeable, rendering it suitable for parallel and distributed computing environments
Measuring Distributional Shifts in Text: The Advantage of Language Model-Based Embeddings
An essential part of monitoring machine learning models in production is
measuring input and output data drift. In this paper, we present a system for
measuring distributional shifts in natural language data and highlight and
investigate the potential advantage of using large language models (LLMs) for
this problem. Recent advancements in LLMs and their successful adoption in
different domains indicate their effectiveness in capturing semantic
relationships for solving various natural language processing problems. The
power of LLMs comes largely from the encodings (embeddings) generated in the
hidden layers of the corresponding neural network. First we propose a
clustering-based algorithm for measuring distributional shifts in text data by
exploiting such embeddings. Then we study the effectiveness of our approach
when applied to text embeddings generated by both LLMs and classical embedding
algorithms. Our experiments show that general-purpose LLM-based embeddings
provide a high sensitivity to data drift compared to other embedding methods.
We propose drift sensitivity as an important evaluation metric to consider when
comparing language models. Finally, we present insights and lessons learned
from deploying our framework as part of the Fiddler ML Monitoring platform over
a period of 18 months