Estimating Mixture Entropy with Pairwise Distances

Mixture distributions arise in many parametric and non-parametric settings -- for example, in Gaussian mixture models and in non-parametric estimation. It is often necessary to compute the entropy of a mixture, but, in most cases, this quantity has no closed-form expression, making some form of approximation necessary. We propose a family of estimators based on a pairwise distance function between mixture components, and show that this estimator class has many attractive properties. For many distributions of interest, the proposed estimators are efficient to compute, differentiable in the mixture parameters, and become exact when the mixture components are clustered. We prove this family includes lower and upper bounds on the mixture entropy. The Chernoff $\alpha$-divergence gives a lower bound when chosen as the distance function, with the Bhattacharyya distance providing the tightest lower bound for components that are symmetric and members of a location family. The Kullback-Leibler divergence gives an upper bound when used as the distance function. We provide closed-form expressions of these bounds for mixtures of Gaussians, and discuss their applications to the estimation of mutual information. We then demonstrate that our bounds are significantly tighter than well-known existing bounds using numeric simulations. This estimator class is very useful in optimization problems involving maximization/minimization of entropy and mutual information, such as MaxEnt and rate distortion problems.Comment: Corrects several errata in published version, in particular in Section V (bounds on mutual information

The Global Joint Distribution of Income and Health

We investigate the evolution of global welfare in two dimensions: income per capita and life expectancy. First, we estimate the marginal distributions of income and life expectancy separately. More importantly, in contrast to previous univariate approaches, we consider income and life expectancy jointly and estimate their bivariate global distribution for 137 countries during 1970 - 2000. We reach several conclusions: the global joint distribution has evolved from a bimodal into a unimodal one, the evolution of the health distribution has preceded that of income, global inequality and poverty has decreased over time and the evolution of the global distribution has been welfare improving. Our decomposition of overall welfare indicates that global inequality would be underestimated if within-country inequality is not taken into account. Moreover, global inequality and poverty would be substantially underestimated if the dependence between the income and health distributions is ignored.income, health, global distribution, inequality, poverty

The quantification of SIMS depth profiles by Maximum Entropy reconstruction

The quantification procedures applied to raw SIMS data were devised on the basis of a simple model for the sputtering and ionisation that occur during measurement. The model and the associated quantification procedures have long been known to be inaccurate. If SIMS is to remain a useful analysis tool in the future, the quantification procedures must be adjusted such that current features of interest are accurately measured. This thesis describes the development of a more accurate (though empirical) model for the effects of the analysis, using the convolution integral. We propose a method for the quantification of SIMS depth profiles appropriate to this model, using Maximum Entropy (MaxEnt) reconstruction. SIMS depth profile data differ significantly from previous applications of the MaxEnt method: the very high signal to background ratio of the technique has lead users to plot the results on a logarithmic axis, giving much importance to extremely small signals. The noise on SIMS depth profiles has been characterised. A number of optimisation algorithms have been developed and tested, and the performance of the MaxEnt method on SIMS data has been assessed. A novel form of the entropy, particularly suited to SIMS depth profiles, has been suggested. This form has given excellent results

Bounds on mutual information of mixture data for classification tasks

The data for many classification problems, such as pattern and speech recognition, follow mixture distributions. To quantify the optimum performance for classification tasks, the Shannon mutual information is a natural information-theoretic metric, as it is directly related to the probability of error. The mutual information between mixture data and the class label does not have an analytical expression, nor any efficient computational algorithms. We introduce a variational upper bound, a lower bound, and three estimators, all employing pair-wise divergences between mixture components. We compare the new bounds and estimators with Monte Carlo stochastic sampling and bounds derived from entropy bounds. To conclude, we evaluate the performance of the bounds and estimators through numerical simulations

Uncertainty Quantification of the Responses of Transient Electromagnetic Disturbance Coupling to Transmission Lines Based on Stochastic Models

L'abstract è presente nell'allegato / the abstract is in the attachmen

Efficient Computational Methods for Structural Reliability and Global Sensitivity Analyses

Uncertainty analysis of a system response is an important part of engineering probabilistic analysis. Uncertainty analysis includes: (a) to evaluate moments of the response; (b) to evaluate reliability analysis of the system; (c) to assess the complete probability distribution of the response; (d) to conduct the parametric sensitivity analysis of the output. The actual model of system response is usually a high-dimensional function of input variables. Although Monte Carlo simulation is a quite general approach for this purpose, it may require an inordinate amount of resources to achieve an acceptable level of accuracy. Development of a computationally efficient method, hence, is of great importance. First of all, the study proposed a moment method for uncertainty quantification of structural systems. However, a key departure is the use of fractional moment of response function, as opposed to integer moment used so far in literature. The advantage of using fractional moment over integer moment was illustrated from the relation of one fractional moment with a couple of integer moments. With a small number of samples to compute the fractional moments, a system output distribution was estimated with the principle of maximum entropy (MaxEnt) in conjunction with the constraints specified in terms of fractional moments. Compared to the classical MaxEnt, a novel feature of the proposed method is that fractional exponent of the MaxEnt distribution is determined through the entropy maximization process, instead of assigned by an analyst in prior. To further minimize the computational cost of the simulation-based entropy method, a multiplicative dimensional reduction method (M-DRM) was proposed to compute the fractional (integer) moments of a generic function with multiple input variables. The M-DRM can accurately approximate a high-dimensional function as the product of a series low-dimensional functions. Together with the principle of maximum entropy, a novel computational approach was proposed to assess the complete probability distribution of a system output. Accuracy and efficiency of the proposed method for structural reliability analysis were verified by crude Monte Carlo simulation of several examples. Application of M-DRM was further extended to the variance-based global sensitivity analysis of a system. Compared to the local sensitivity analysis, the variance-based sensitivity index can provide significance information about an input random variable. Since each component variance is defined as a conditional expectation with respect to the system model function, the separable nature of the M-DRM approximation can simplify the high-dimension integrations in sensitivity analysis. Several examples were presented to illustrate the numerical accuracy and efficiency of the proposed method in comparison to the Monte Carlo simulation method. The last contribution of the proposed study is the development of a computationally efficient method for polynomial chaos expansion (PCE) of a system's response. This PCE model can be later used uncertainty analysis. However, evaluation of coefficients of a PCE meta-model is computational demanding task due to the involved high-dimensional integrations. With the proposed M-DRM, the involved computational cost can be remarkably reduced compared to the classical methods in literature (simulation method or tensor Gauss quadrature method). Accuracy and efficiency of the proposed method for polynomial chaos expansion were verified by considering several practical examples.1 yea

Investigations on the properties and estimation of earth response operators from EM sounding data

Incl. 3 reprints at backAvailable from British Library Document Supply Centre- DSC:D82993 / BLDSC - British Library Document Supply CentreSIGLEGBUnited Kingdo