A note on entropy estimation

We compare an entropy estimator $H_z$ recently discussed in [10] with two estimators $H_1$ and $H_2$ introduced in [6][7]. We prove the identity $H_z \equiv H_1$, which has not been taken into account in [10]. Then, we prove that the statistical bias of $H_1$ is less than the bias of the ordinary likelihood estimator of entropy. Finally, by numerical simulation we verify that for the most interesting regime of small sample estimation and large event spaces, the estimator $H_2$ has a significant smaller statistical error than $H_z$.Comment: 7 pages, including 4 figures; two references adde

The asymptotic minimax risk for the estimation of constrained binomial and multinomial probabilities

In this note we present a direct and simple approach to obtain bounds on the asymptotic minimax risk for the estimation of restrained binominal and multinominal proportions. Quadratic, normalized quadratic and entropy loss are considered and it is demonstrated that in all cases linear estimators are asymptotically minimax optimal. For the quadratic loss function the asymptotic minimax rsik does not change unless a neighborhood of the point 1/2 is excluded by the restrictions on the parameter space. For the two other loss functions the asymptotic minimax risks remain unchanged if additional knowledge about the location of the unknown probability of success is imposed. The results are also extended to the problem of minimax estimation of a vector of contrained multinominal propabilities. --binominal distribution,multinominal distribution,entropy loss,quadratic loss,constrained parameter space,least favourable distribution

Transfer Entropy Estimation and Directional Coupling Change Detection in Biomedical Time Series

Background: The detection of change in magnitude of directional coupling between two non-linear time series is a common subject of interest in the biomedical domain, including studies involving the respiratory chemoreflex system. Although transfer entropy is a useful tool in this avenue, no study to date has investigated how different transfer entropy estimation methods perform in typical biomedical applications featuring small sample size and presence of outliers. Methods: With respect to detection of increased coupling strength, we compared three transfer entropy estimation techniques using both simulated time series and respiratory recordings from lambs. The following estimation methods were analyzed: fixed-binning with ranking, kernel density estimation (KDE), and the Darbellay-Vajda (D-V) adaptive partitioning algorithm extended to three dimensions. In the simulated experiment, sample size was varied from 50 to 200, while coupling strength was increased. In order to introduce outliers, the heavy-tailed Laplace distribution was utilized. In the lamb experiment, the objective was to detect increased respiratoryrelated chemosensitivity to O[subscript 2] and CO[subscript 2] induced by a drug, domperidone. Specifically, the separate influence of end-tidal PO[subscript 2] and PCO[subscript 2] on minute ventilation ([dot over V][subscript E]) before and after administration of domperidone was analyzed. Results: In the simulation, KDE detected increased coupling strength at the lowest SNR among the three methods. In the lamb experiment, D-V partitioning resulted in the statistically strongest increase in transfer entropy post-domperidone for PO2 → [dot over V][subscript E]. In addition, D-V partitioning was the only method that could detect an increase in transfer entropy for PCO[subscript 2] → [dot over V][subscript E], in agreement with experimental findings. Conclusions: Transfer entropy is capable of detecting directional coupling changes in non-linear biomedical time series analysis featuring a small number of observations and presence of outliers. The results of this study suggest that fixed-binning, even with ranking, is too primitive, and although there is no clear winner between KDE and D-V partitioning, the reader should note that KDE requires more computational time and extensive parameter selection than D-V partitioning. We hope this study provides a guideline for selection of an appropriate transfer entropy estimation method.National Institutes of Health (U.S.) (Grant R01-EB001659)National Institutes of Health (U.S.) (Grant R01- HL73146)National Institutes of Health (U.S.) (Grant HL085188-01A2)National Institutes of Health (U.S.) (Grant HL090897-01A2)National Institutes of Health (U.S.) (Grant K24 HL093218-01A1)National Institutes of Health (U.S.) (Cooperative Agreement U01-EB-008577)National Institutes of Health (U.S.) (Training Grant T32-HL07901))American Heart Association (Grant 0840159N

Statistical computation of Boltzmann entropy and estimation of the optimal probability density function from statistical sample

In this work, we investigate the statistical computation of the Boltzmann entropy of statistical samples. For this purpose, we use both histogram and kernel function to estimate the probability density function of statistical samples. We find that, due to coarse-graining, the entropy is a monotonic increasing function of the bin width for histogram or bandwidth for kernel estimation, which seems to be difficult to select an optimal bin width/bandwidth for computing the entropy. Fortunately, we notice that there exists a minimum of the first derivative of entropy for both histogram and kernel estimation, and this minimum point of the first derivative asymptotically points to the optimal bin width or bandwidth. We have verified these findings by large amounts of numerical experiments. Hence, we suggest that the minimum of the first derivative of entropy be used as a selector for the optimal bin width or bandwidth of density estimation. Moreover, the optimal bandwidth selected by the minimum of the first derivative of entropy is purely data-based, independent of the unknown underlying probability density distribution, which is obviously superior to the existing estimators. Our results are not restricted to one-dimensional, but can also be extended to multivariate cases. It should be emphasized, however, that we do not provide a robust mathematical proof of these findings, and we leave these issues with those who are interested in them.Comment: 8 pages, 6 figures, MNRAS, in the pres

A Maximum Entropy Procedure to Solve Likelihood Equations

In this article, we provide initial findings regarding the problem of solving likelihood equations by means of a maximum entropy (ME) approach. Unlike standard procedures that require equating the score function of the maximum likelihood problem at zero, we propose an alternative strategy where the score is instead used as an external informative constraint to the maximization of the convex Shannon\u2019s entropy function. The problem involves the reparameterization of the score parameters as expected values of discrete probability distributions where probabilities need to be estimated. This leads to a simpler situation where parameters are searched in smaller (hyper) simplex space. We assessed our proposal by means of empirical case studies and a simulation study, the latter involving the most critical case of logistic regression under data separation. The results suggested that the maximum entropy reformulation of the score problem solves the likelihood equation problem. Similarly, when maximum likelihood estimation is difficult, as is the case of logistic regression under separation, the maximum entropy proposal achieved results (numerically) comparable to those obtained by the Firth\u2019s bias-corrected approach. Overall, these first findings reveal that a maximum entropy solution can be considered as an alternative technique to solve the likelihood equation