18 research outputs found
Reduced perplexity: Uncertainty measures without entropy
Conference paper presented at Recent Advances in Info-Metrics, Washington, DC, 2014. Under review for a book chapter in "Recent innovations in info-metrics: a cross-disciplinary perspective on information and information processing" by Oxford University Press.A simple, intuitive approach to the assessment of probabilistic inferences is introduced. The Shannon information metrics are translated to the probability domain. The translation shows that the negative logarithmic score and the geometric mean are equivalent measures of the accuracy of a probabilistic inference. Thus there is both a quantitative reduction in perplexity as good inference algorithms reduce the uncertainty and a qualitative reduction due to the increased clarity between the original set of inferences and their average, the geometric mean. Further insight is provided by showing that the Renyi and Tsallis entropy functions translated to the probability domain are both the weighted generalized mean of the distribution. The generalized mean of probabilistic inferences forms a Risk Profile of the performance. The arithmetic mean is used to measure the decisiveness, while the -2/3 mean is used to measure the robustness
Assessing probabilistic inference by comparing the generalized mean of the model and source probabilities
An approach to the assessment of probabilistic inference is described which quantifies the performance on the probability scale. From both information and Bayesian theory, the central tendency of an inference is proven to be the geometric mean of the probabilities reported for the actual outcome and is referred to as the “Accuracy”. Upper and lower error bars on the accuracy are provided by the arithmetic mean and the −2/3 mean. The arithmetic is called the “Decisiveness” due to its similarity with the cost of a decision and the −2/3 mean is called the “Robustness”, due to its sensitivity to outlier errors. Visualization of inference performance is facilitated by plotting the reported model probabilities versus the histogram calculated source probabilities. The visualization of the calibration between model and source is summarized on both axes by the arithmetic, geometric, and −2/3 means. From information theory, the performance of the inference is related to the cross-entropy between the model and source distribution. Just as cross-entropy is the sum of the entropy and the divergence; the accuracy of a model can be decomposed into a component due to the source uncertainty and the divergence between the source and model. Translated to the probability domain these quantities are plotted as the average model probability versus the average source probability. The divergence probability is the average model probability divided by the average source probability. When an inference is over/under-confident, the arithmetic mean of the model increases/decreases, while the −2/3 mean decreases/increases, respectively.https://doi.org/10.3390/e19060286Published versio
On the average uncertainty for systems with nonlinear coupling
The increased uncertainty and complexity of nonlinear systems have motivated
investigators to consider generalized approaches to defining an entropy
function. New insights are achieved by defining the average uncertainty in the
probability domain as a transformation of entropy functions. The Shannon
entropy when transformed to the probability domain is the weighted geometric
mean of the probabilities. For the exponential and Gaussian distributions, we
show that the weighted geometric mean of the distribution is equal to the
density of the distribution at the location plus the scale, i.e. at the width
of the distribution. The average uncertainty is generalized via the weighted
generalized mean, in which the moment is a function of the nonlinear source.
Both the Renyi and Tsallis entropies transform to this definition of the
generalized average uncertainty in the probability domain. For the generalized
Pareto and Student's t-distributions, which are the maximum entropy
distributions for these generalized entropies, the appropriate weighted
generalized mean also equals the density of the distribution at the location
plus scale. A coupled entropy function is proposed, which is equal to the
normalized Tsallis entropy divided by one plus the coupling.Comment: 24 pages, including 4 figures and 1 tabl
Use of the geometric mean as a statistic for the scale of the coupled Gaussian distributions
The geometric mean is shown to be an appropriate statistic for the scale of a
heavy-tailed coupled Gaussian distribution or equivalently the Student's t
distribution. The coupled Gaussian is a member of a family of distributions
parameterized by the nonlinear statistical coupling which is the reciprocal of
the degree of freedom and is proportional to fluctuations in the inverse scale
of the Gaussian. Existing estimators of the scale of the coupled Gaussian have
relied on estimates of the full distribution, and they suffer from problems
related to outliers in heavy-tailed distributions. In this paper, the scale of
a coupled Gaussian is proven to be equal to the product of the generalized mean
and the square root of the coupling. From our numerical computations of the
scales of coupled Gaussians using the generalized mean of random samples, it is
indicated that only samples from a Cauchy distribution (with coupling parameter
one) form an unbiased estimate with diminishing variance for large samples.
Nevertheless, we also prove that the scale is a function of the geometric mean,
the coupling term and a harmonic number. Numerical experiments show that this
estimator is unbiased with diminishing variance for large samples for a broad
range of coupling values.Comment: 17 pages, 5 figure
A risk profile for information fusion algorithms
E.T. Jaynes, originator of the maximum entropy interpretation of statistical
mechanics, emphasized that there is an inevitable trade-off between the
conflicting requirements of robustness and accuracy for any inferencing
algorithm. This is because robustness requires discarding of information in
order to reduce the sensitivity to outliers. The principal of nonlinear
statistical coupling, which is an interpretation of the Tsallis entropy
generalization, can be used to quantify this trade-off. The coupled-surprisal,
-ln_k (p)=-(p^k-1)/k, is a generalization of Shannon surprisal or the
logarithmic scoring rule, given a forecast p of a true event by an inferencing
algorithm. The coupling parameter k=1-q, where q is the Tsallis entropy index,
is the degree of nonlinear coupling between statistical states. Positive
(negative) values of nonlinear coupling decrease (increase) the surprisal
information metric and thereby biases the risk in favor of decisive (robust)
algorithms relative to the Shannon surprisal (k=0). We show that translating
the average coupled-surprisal to an effective probability is equivalent to
using the generalized mean of the true event probabilities as a scoring rule.
The metric is used to assess the robustness, accuracy, and decisiveness of a
fusion algorithm. We use a two-parameter fusion algorithm to combine input
probabilities from N sources. The generalized mean parameter 'alpha' varies the
degree of smoothing and raising to a power N^beta with beta between 0 and 1
provides a model of correlation.Comment: 15 pages, 4 figure