Origins of the Combinatorial Basis of Entropy

The combinatorial basis of entropy, given by Boltzmann, can be written $H = N^{-1} \ln \mathbb{W}$, where $H$ is the dimensionless entropy, $N$ is the number of entities and $\mathbb{W}$ is number of ways in which a given realization of a system can occur (its statistical weight). This can be broadened to give generalized combinatorial (or probabilistic) definitions of entropy and cross-entropy: $H=\kappa (\phi(\mathbb{W}) +C)$ and $D=-\kappa (\phi(\mathbb{P}) +C)$, where $\mathbb{P}$ is the probability of a given realization, $\phi$ is a convenient transformation function, $\kappa$ is a scaling parameter and $C$ an arbitrary constant. If $\mathbb{W}$ or $\mathbb{P}$ satisfy the multinomial weight or distribution, then using $\phi(\cdot)=\ln(\cdot)$ and $\kappa=N^{-1}$, $H$ and $D$ asymptotically converge to the Shannon and Kullback-Leibler functions. In general, however, $\mathbb{W}$ or $\mathbb{P}$ need not be multinomial, nor may they approach an asymptotic limit. In such cases, the entropy or cross-entropy function can be {\it defined} so that its extremization ("MaxEnt'' or "MinXEnt"), subject to the constraints, gives the ``most probable'' (``MaxProb'') realization of the system. This gives a probabilistic basis for MaxEnt and MinXEnt, independent of any information-theoretic justification. This work examines the origins of the governing distribution $\mathbb{P}$.... (truncated)Comment: MaxEnt07 manuscript, version 4 revise

Combinatorial Information Theory: I. Philosophical Basis of Cross-Entropy and Entropy

This study critically analyses the information-theoretic, axiomatic and combinatorial philosophical bases of the entropy and cross-entropy concepts. The combinatorial basis is shown to be the most fundamental (most primitive) of these three bases, since it gives (i) a derivation for the Kullback-Leibler cross-entropy and Shannon entropy functions, as simplified forms of the multinomial distribution subject to the Stirling approximation; (ii) an explanation for the need to maximize entropy (or minimize cross-entropy) to find the most probable realization; and (iii) new, generalized definitions of entropy and cross-entropy - supersets of the Boltzmann principle - applicable to non-multinomial systems. The combinatorial basis is therefore of much broader scope, with far greater power of application, than the information-theoretic and axiomatic bases. The generalized definitions underpin a new discipline of ``{\it combinatorial information theory}'', for the analysis of probabilistic systems of any type. Jaynes' generic formulation of statistical mechanics for multinomial systems is re-examined in light of the combinatorial approach. (abbreviated abstract)Comment: 45 pp; 1 figure; REVTex; updated version 5 (incremental changes

Measure Recognition Problem

This is an article in mathematics, specifically in set theory. On the example of the Measure Recognition Problem (MRP) the article highlights the phenomenon of the utility of a multidisciplinary mathematical approach to a single mathematical problem, in particular the value of a set-theoretic analysis. MRP asks if for a given Boolean algebra \algB and a property $\Phi$ of measures one can recognize by purely combinatorial means if \algB supports a strictly positive measure with property $\Phi$. The most famous instance of this problem is MRP(countable additivity), and in the first part of the article we survey the known results on this and some other problems. We show how these results naturally lead to asking about two other specific instances of the problem MRP, namely MRP(nonatomic) and MRP(separable). Then we show how our recent work D\v zamonja and Plebanek (2006) gives an easy solution to the former of these problems, and gives some partial information about the latter. The long term goal of this line of research is to obtain a structure theory of Boolean algebras that support a finitely additive strictly positive measure, along the lines of Maharam theorem which gives such a structure theorem for measure algebras

Game-theoretic Resource Allocation Methods for Device-to-Device (D2D) Communication

Device-to-device (D2D) communication underlaying cellular networks allows mobile devices such as smartphones and tablets to use the licensed spectrum allocated to cellular services for direct peer-to-peer transmission. D2D communication can use either one-hop transmission (i.e., in D2D direct communication) or multi-hop cluster-based transmission (i.e., in D2D local area networks). The D2D devices can compete or cooperate with each other to reuse the radio resources in D2D networks. Therefore, resource allocation and access for D2D communication can be treated as games. The theories behind these games provide a variety of mathematical tools to effectively model and analyze the individual or group behaviors of D2D users. In addition, game models can provide distributed solutions to the resource allocation problems for D2D communication. The aim of this article is to demonstrate the applications of game-theoretic models to study the radio resource allocation issues in D2D communication. The article also outlines several key open research directions.Comment: Accepted. IEEE Wireless Comms Mag. 201

Adaptive group testing as channel coding with feedback

Group testing is the combinatorial problem of identifying the defective items in a population by grouping items into test pools. Recently, nonadaptive group testing - where all the test pools must be decided on at the start - has been studied from an information theory point of view. Using techniques from channel coding, upper and lower bounds have been given on the number of tests required to accurately recover the defective set, even when the test outcomes can be noisy. In this paper, we give the first information theoretic result on adaptive group testing - where the outcome of previous tests can influence the makeup of future tests. We show that adaptive testing does not help much, as the number of tests required obeys the same lower bound as nonadaptive testing. Our proof uses similar techniques to the proof that feedback does not improve channel capacity.Comment: 4 pages, 1 figur