Computing Entropies With Nested Sampling

The Shannon entropy, and related quantities such as mutual information, can be used to quantify uncertainty and relevance. However, in practice, it can be difficult to compute these quantities for arbitrary probability distributions, particularly if the probability mass functions or densities cannot be evaluated. This paper introduces a computational approach, based on Nested Sampling, to evaluate entropies of probability distributions that can only be sampled. I demonstrate the method on three examples: a simple gaussian example where the key quantities are available analytically; (ii) an experimental design example about scheduling observations in order to measure the period of an oscillating signal; and (iii) predicting the future from the past in a heavy-tailed scenario.Comment: Accepted for publication in Entropy. 21 pages, 3 figures. Software available at https://github.com/eggplantbren/InfoNes

The Implications of the Early Formation of Life on Earth

One of the most interesting unsolved questions in science today is the question of life on other planets. At the present time it is safe to say that we do not have much of an idea as to whether life is common or exceedingly rare in the universe, and this will probably not be solved for certain unless definitive evidence of extraterrestrial life is found in the future. Our presence on Earth is just as consistent with the hypothesis that life is extremely rare as it is with the hypothesis that it is common, since if there was only one planet with intelligent life, we would find ourselves on it. However, we have more information than this, such as the the surprisingly short length of time it took for life to arise on Earth. Previous authors have analysed this information, concluding that it is evidence that the probability of abiogenesis is moderate ($>$ 13% with 95% probability) and cannot be extremely small. In this paper I use simple probabilistic model to show that this conclusion was based more on an unintentional assumption than on the data. While the early formation of life on Earth provides some evidence in the direction of life being common, it is far from conclusive, and in particular does not rule out the possibility that abiogenesis has only occurred once in the history of the universe.Comment: Submitted, but seems to have fallen into a black hole sinc

Inference for Trans-dimensional Bayesian Models with Diffusive Nested Sampling

Many inference problems involve inferring the number $N$ of components in some region, along with their properties $\{\mathbf{x}_i\}_{i=1}^N$, from a dataset $\mathcal{D}$. A common statistical example is finite mixture modelling. In the Bayesian framework, these problems are typically solved using one of the following two methods: i) by executing a Monte Carlo algorithm (such as Nested Sampling) once for each possible value of $N$, and calculating the marginal likelihood or evidence as a function of $N$; or ii) by doing a single run that allows the model dimension $N$ to change (such as Markov Chain Monte Carlo with birth/death moves), and obtaining the posterior for $N$ directly. In this paper we present a general approach to this problem that uses trans-dimensional MCMC embedded within a Nested Sampling algorithm, allowing us to explore the posterior distribution and calculate the marginal likelihood (summed over $N$) even if the problem contains a phase transition or other difficult features such as multimodality. We present two example problems, finding sinusoidal signals in noisy data, and finding and measuring galaxies in a noisy astronomical image. Both of the examples demonstrate phase transitions in the relationship between the likelihood and the cumulative prior mass, highlighting the need for Nested Sampling.Comment: Only published here for the time being. 17 pages, 10 figures. Software available at https://github.com/eggplantbren/RJObjec