184 research outputs found
Inductive Conformal Martingales for Change-Point Detection
We consider the problem of quickest change-point detection in data streams.
Classical change-point detection procedures, such as CUSUM, Shiryaev-Roberts
and Posterior Probability statistics, are optimal only if the change-point
model is known, which is an unrealistic assumption in typical applied problems.
Instead we propose a new method for change-point detection based on Inductive
Conformal Martingales, which requires only the independence and identical
distribution of observations. We compare the proposed approach to standard
methods, as well as to change-point detection oracles, which model a typical
practical situation when we have only imprecise (albeit parametric) information
about pre- and post-change data distributions. Results of comparison provide
evidence that change-point detection based on Inductive Conformal Martingales
is an efficient tool, capable to work under quite general conditions unlike
traditional approaches.Comment: 22 pages, 9 figures, 5 table
Testing randomness online
The hypothesis of randomness is fundamental in statistical machine learning
and in many areas of nonparametric statistics; it says that the observations
are assumed to be independent and coming from the same unknown probability
distribution. This hypothesis is close, in certain respects, to the hypothesis
of exchangeability, which postulates that the distribution of the observations
is invariant with respect to their permutations. This paper reviews known
methods of testing the two hypotheses concentrating on the online mode of
testing, when the observations arrive sequentially. All known online methods
for testing these hypotheses are based on conformal martingales, which are
defined and studied in detail. The paper emphasizes conceptual and practical
aspects and states two kinds of results. Validity results limit the probability
of a false alarm or the frequency of false alarms for various procedures based
on conformal martingales, including conformal versions of the CUSUM and
Shiryaev-Roberts procedures. Efficiency results establish connections between
randomness, exchangeability, and conformal martingales.Comment: 34 pages, 1 table, 4 figure
Online Distribution Shift Detection via Recency Prediction
When deploying modern machine learning-enabled robotic systems in high-stakes
applications, detecting distribution shift is critical. However, most existing
methods for detecting distribution shift are not well-suited to robotics
settings, where data often arrives in a streaming fashion and may be very
high-dimensional. In this work, we present an online method for detecting
distribution shift with guarantees on the false positive rate - i.e., when
there is no distribution shift, our system is very unlikely (with probability
) to falsely issue an alert; any alerts that are issued should
therefore be heeded. Our method is specifically designed for efficient
detection even with high dimensional data, and it empirically achieves up to
11x faster detection on realistic robotics settings compared to prior work
while maintaining a low false negative rate in practice (whenever there is a
distribution shift in our experiments, our method indeed emits an alert). We
demonstrate our approach in both simulation and hardware for a visual servoing
task, and show that our method indeed issues an alert before a failure occurs
Anomalous Edge Detection in Edge Exchangeable Social Network Models
This paper studies detecting anomalous edges in directed graphs that model
social networks. We exploit edge exchangeability as a criterion for
distinguishing anomalous edges from normal edges. Then we present an anomaly
detector based on conformal prediction theory; this detector has a guaranteed
upper bound for false positive rate. In numerical experiments, we show that the
proposed algorithm achieves superior performance to baseline methods
Game-theoretic statistics and safe anytime-valid inference
Safe anytime-valid inference (SAVI) provides measures of statistical evidence
and certainty -- e-processes for testing and confidence sequences for
estimation -- that remain valid at all stopping times, accommodating continuous
monitoring and analysis of accumulating data and optional stopping or
continuation for any reason. These measures crucially rely on test martingales,
which are nonnegative martingales starting at one. Since a test martingale is
the wealth process of a player in a betting game, SAVI centrally employs
game-theoretic intuition, language and mathematics. We summarize the SAVI goals
and philosophy, and report recent advances in testing composite hypotheses and
estimating functionals in nonparametric settings.Comment: 25 pages. Under review. ArXiv does not compile/space some references
properl
Game-theoretic statistics and safe anytime-valid inference
Safe anytime-valid inference (SAVI) provides measures of statistical evidence and certainty—e-processes for testing and confidence sequences for estimation—that remain valid at all stopping times, accommodating continuous monitoring and analysis of accumulating data and optional stopping or continuation for any reason. These measures crucially rely on test martingales, which are nonnegative martingales starting at one. Since a test martingale is the wealth process of a player in a betting game, SAVI centrally employs game-theoretic intuition, language and mathematics. We summarize the SAVI goals and philosophy, and report recent advances in testing composite hypotheses and estimating functionals in nonparametric settings
- …