How biased are maximum entropy models?

Maximum entropy models have become popular statistical models in neuroscience and other areas in biology, and can be useful tools for obtaining estimates of mutual information in biological systems. However, maximum entropy models fit to small data sets can be subject to sampling bias; i.e. the true entropy of the data can be severely underestimated. Here we study the sampling properties of estimates of the entropy obtained from maximum entropy models. We show that if the data is generated by a distribution that lies in the model class, the bias is equal to the number of parameters divided by twice the number of observations. However, in practice, the true distribution is usually outside the model class, and we show here that this misspecification can lead to much larger bias. We provide a perturbative approximation of the maximally expected bias when the true model is out of model class, and we illustrate our results using numerical simulations of an Ising model; i.e. the second-order maximum entropy distribution on binary data.

Asymptotic consistency under large entropy sampling designs with unequal probabilities

A large part of survey sampling literature is devoted to unequal probabilities sampling designs without replacement. Brewer and Hanif (1983) provided a summary of these sampling designs. The maximum entropy designs is one of them. Consistency results have been proven for the maximum entropy sampling (Hájek, 1964). The aim is to give sufficient conditions under which Hájek (1964) consistency results still hold for large entropy sampling designs which are different from the maximum entropy design. These conditions involve modes of convergence of sampling designs towards the maximum entropy design. We show that these conditions are satisfied for the popular Rao-Sampford (Rao, 1965, Sampford, 1967) design. Our consistency results are applied to the Hájek (1964) simple variance estimator. This estimator does not require joint-inclusion probabilities and can be easily estimated using weighted least squares regression (Berger, 2004, 2005b). Deville (1999) conjectured that this estimator is suitable for any sampling designs (see also Brewer and Donadio, 2003). Our consistency result gives regularity conditions under which this estimator is consistent which justifies Deville’s (1999) conjecture

Maximum entropy properties of discrete-time first-order stable spline kernel

The first order stable spline (SS-1) kernel is used extensively in regularized system identification. In particular, the stable spline estimator models the impulse response as a zero-mean Gaussian process whose covariance is given by the SS-1 kernel. In this paper, we discuss the maximum entropy properties of this prior. In particular, we formulate the exact maximum entropy problem solved by the SS-1 kernel without Gaussian and uniform sampling assumptions. Under general sampling schemes, we also explicitly derive the special structure underlying the SS-1 kernel (e.g. characterizing the tridiagonal nature of its inverse), also giving to it a maximum entropy covariance completion interpretation. Along the way similar maximum entropy properties of the Wiener kernel are also given

TRUNCATED REGRESSION IN EMPIRICAL ESTIMATION

In this paper we illustrate the use of alternative truncated regression estimators for the general linear model. These include variations of maximum likelihood, Bayesian, and maximum entropy estimators in which the error distributions are doubly truncated. To evaluate the performance of the estimators (e.g., efficiency) for a range of sample sizes, Monte Carlo sampling experiments are performed. We then apply each estimator to a factor demand equation for wheat-by-class.doubly truncated samples, Bayesian regression, maximum entropy, wheat-by-class, Research Methods/ Statistical Methods,