Estimating probabilities from experimental frequencies

Estimating the probability distribution 'q' governing the behaviour of a certain variable by sampling its value a finite number of times most typically involves an error. Successive measurements allow the construction of a histogram, or frequency count 'f', of each of the possible outcomes. In this work, the probability that the true distribution be 'q', given that the frequency count 'f' was sampled, is studied. Such a probability may be written as a Gibbs distribution. A thermodynamic potential, which allows an easy evaluation of the mean Kullback-Leibler divergence between the true and measured distribution, is defined. For a large number of samples, the expectation value of any function of 'q' is expanded in powers of the inverse number of samples. As an example, the moments, the entropy and the mutual information are analyzed.Comment: 10 pages, 3 figures, to be published in Physical Review

BLIND SOURCE SEPARATION USING MAXIMUM ENTROPY PDF ESTIMATION BASED ON FRACTIONAL MOMENTS

Abstract. Recovering a set of independent sources which are linearly mixed is the main task of the blind source separation. Utilizing different methods such as infomax principle, mutual information and maximum likelihood leads to simple iterative procedures such as natural gradient algorithms. These algorithms depend on a nonlinear function (known as score or activation function) of source distributions. Since there is no prior knowledge of source distributions, the optimality of the algorithms is based on the choice of a suitable parametric density model. In this paper, we propose an adaptive optimal score function based on the fractional moments of the sources. In order to obtain a parametric model for the source distributions, we use a few sampled fractional moments to construct the maximum entropy probability density function (PDF) estimation . By applying an optimization method we can obtain the optimal fractional moments that best fit the source distributions. Using the fractional moments (FM) instead of the integer moments causes the maximum entropy estimated PDF to converge to the true PDF much faster . The simulation results show that unlike the most previous proposed models for the nonlinear score function, which are limited to some sorts of source families such as sub-gaussian and super-gaussian or some forms of source distribution models such as generalized gaussian distribution, our new model achieves better results for every source signal without any prior assumption for its randomness behavior

Bivariate Gamma Distributions for Image Registration and Change Detection

This paper evaluates the potential interest of using bivariate gamma distributions for image registration and change detection. The first part of this paper studies estimators for the parameters of bivariate gamma distributions based on the maximum likelihood principle and the method of moments. The performance of both methods are compared in terms of estimated mean square errors and theoretical asymptotic variances. The mutual information is a classical similarity measure which can be used for image registration or change detection. The second part of the paper studies some properties of the mutual information for bivariate Gamma distributions. Image registration and change detection techniques based on bivariate gamma distributions are finally investigated. Simulation results conducted on synthetic and real data are very encouraging. Bivariate gamma distributions are good candidates allowing us to develop new image registration algorithms and new change detectors

On the Outage Capacity of Correlated Multiple-Path MIMO Channels

The use of multi-antenna arrays in both transmission and reception has been shown to dramatically increase the throughput of wireless communication systems. As a result there has been considerable interest in characterizing the ergodic average of the mutual information for realistic correlated channels. Here, an approach is presented that provides analytic expressions not only for the average, but also the higher cumulant moments of the distribution of the mutual information for zero-mean Gaussian (multiple-input multiple-output) MIMO channels with the most general multipath covariance matrices when the channel is known at the receiver. These channels include multi-tap delay paths, as well as general channels with covariance matrices that cannot be written as a Kronecker product, such as dual-polarized antenna arrays with general correlations at both transmitter and receiver ends. The mathematical methods are formally valid for large antenna numbers, in which limit it is shown that all higher cumulant moments of the distribution, other than the first two scale to zero. Thus, it is confirmed that the distribution of the mutual information tends to a Gaussian, which enables one to calculate the outage capacity. These results are quite accurate even in the case of a few antennas, which makes this approach applicable to realistic situations.Comment: submitted for publication IEEE Trans. Information Theory; IEEEtran documentstyl

Asymptotic Performance of Linear Receivers in MIMO Fading Channels

Linear receivers are an attractive low-complexity alternative to optimal processing for multi-antenna MIMO communications. In this paper we characterize the information-theoretic performance of MIMO linear receivers in two different asymptotic regimes. For fixed number of antennas, we investigate the limit of error probability in the high-SNR regime in terms of the Diversity-Multiplexing Tradeoff (DMT). Following this, we characterize the error probability for fixed SNR in the regime of large (but finite) number of antennas. As far as the DMT is concerned, we report a negative result: we show that both linear Zero-Forcing (ZF) and linear Minimum Mean-Square Error (MMSE) receivers achieve the same DMT, which is largely suboptimal even in the case where outer coding and decoding is performed across the antennas. We also provide an approximate quantitative analysis of the markedly different behavior of the MMSE and ZF receivers at finite rate and non-asymptotic SNR, and show that while the ZF receiver achieves poor diversity at any finite rate, the MMSE receiver error curve slope flattens out progressively, as the coding rate increases. When SNR is fixed and the number of antennas becomes large, we show that the mutual information at the output of a MMSE or ZF linear receiver has fluctuations that converge in distribution to a Gaussian random variable, whose mean and variance can be characterized in closed form. This analysis extends to the linear receiver case a well-known result previously obtained for the optimal receiver. Simulations reveal that the asymptotic analysis captures accurately the outage behavior of systems even with a moderate number of antennas.Comment: 48 pages, Submitted to IEEE Transactions on Information Theor

Performance of MMSE MIMO Receivers: A Large N Analysis for Correlated Channels

Linear receivers are considered as an attractive low-complexity alternative to optimal processing for multi-antenna MIMO communications. In this paper we characterize the performance of MMSE MIMO receivers in the limit of large antenna numbers in the presence of channel correlations. Using the replica method, we generalize our results obtained in arXiv:0810.0883 to Kronecker-product correlated channels and calculate the asymptotic mean and variance of the mutual information of a MIMO system of parallel MMSE subchannels. The replica method allows us to use the ties between the optimal receiver mutual information and the MMSE SIR of Gaussian inputs to calculate the joint moments of the SIRs of the MMSE subchannels. Using the methodology discussed in arXiv:0810.0883 it can be shown that the mutual information converges in distribution to a Gaussian random variable. Our results agree very well with simulations even with a moderate number of antennas.Comment: Invited article at the IEEE Vehicular Technology Conference, Barcelona 200

Entanglement negativity and conformal field theory: a Monte Carlo study

We investigate the behavior of the moments of the partially transposed reduced density matrix \rho^{T_2}_A in critical quantum spin chains. Given subsystem A as union of two blocks, this is the (matrix) transposed of \rho_A with respect to the degrees of freedom of one of the two. This is also the main ingredient for constructing the logarithmic negativity. We provide a new numerical scheme for calculating efficiently all the moments of \rho_A^{T_2} using classical Monte Carlo simulations. In particular we study several combinations of the moments which are scale invariant at a critical point. Their behavior is fully characterized in both the critical Ising and the anisotropic Heisenberg XXZ chains. For two adjacent blocks we find, in both models, full agreement with recent CFT calculations. For disjoint ones, in the Ising chain finite size corrections are non negligible. We demonstrate that their exponent is the same governing the unusual scaling corrections of the mutual information between the two blocks. Monte Carlo data fully match the theoretical CFT prediction only in the asymptotic limit of infinite intervals. Oppositely, in the Heisenberg chain scaling corrections are smaller and, already at finite (moderately large) block sizes, Monte Carlo data are in excellent agreement with the asymptotic CFT result.Comment: 31 pages, 10 figures. Minor changes, published versio

Locally Orderless Registration

Image registration is an important tool for medical image analysis and is used to bring images into the same reference frame by warping the coordinate field of one image, such that some similarity measure is minimized. We study similarity in image registration in the context of Locally Orderless Images (LOI), which is the natural way to study density estimates and reveals the 3 fundamental scales: the measurement scale, the intensity scale, and the integration scale. This paper has three main contributions: Firstly, we rephrase a large set of popular similarity measures into a common framework, which we refer to as Locally Orderless Registration, and which makes full use of the features of local histograms. Secondly, we extend the theoretical understanding of the local histograms. Thirdly, we use our framework to compare two state-of-the-art intensity density estimators for image registration: The Parzen Window (PW) and the Generalized Partial Volume (GPV), and we demonstrate their differences on a popular similarity measure, Normalized Mutual Information (NMI). We conclude, that complicated similarity measures such as NMI may be evaluated almost as fast as simple measures such as Sum of Squared Distances (SSD) regardless of the choice of PW and GPV. Also, GPV is an asymmetric measure, and PW is our preferred choice.Comment: submitte