14,118 research outputs found
Precision-Recall Curves Using Information Divergence Frontiers
Despite the tremendous progress in the estimation of generative models, the
development of tools for diagnosing their failures and assessing their
performance has advanced at a much slower pace. Recent developments have
investigated metrics that quantify which parts of the true distribution is
modeled well, and, on the contrary, what the model fails to capture, akin to
precision and recall in information retrieval. In this paper, we present a
general evaluation framework for generative models that measures the trade-off
between precision and recall using R\'enyi divergences. Our framework provides
a novel perspective on existing techniques and extends them to more general
domains. As a key advantage, this formulation encompasses both continuous and
discrete models and allows for the design of efficient algorithms that do not
have to quantize the data. We further analyze the biases of the approximations
used in practice.Comment: Updated to the AISTATS 2020 versio
Revisiting Precision and Recall Definition for Generative Model Evaluation
In this article we revisit the definition of Precision-Recall (PR) curves for
generative models proposed by Sajjadi et al. (arXiv:1806.00035). Rather than
providing a scalar for generative quality, PR curves distinguish mode-collapse
(poor recall) and bad quality (poor precision). We first generalize their
formulation to arbitrary measures, hence removing any restriction to finite
support. We also expose a bridge between PR curves and type I and type II error
rates of likelihood ratio classifiers on the task of discriminating between
samples of the two distributions. Building upon this new perspective, we
propose a novel algorithm to approximate precision-recall curves, that shares
some interesting methodological properties with the hypothesis testing
technique from Lopez-Paz et al (arXiv:1610.06545). We demonstrate the interest
of the proposed formulation over the original approach on controlled
multi-modal datasets.Comment: ICML 201
- …