Testing whether a Learning Procedure is Calibrated

A learning procedure takes as input a dataset and performs inference for the parameters $\theta$ of a model that is assumed to have given rise to the dataset. Here we consider learning procedures whose output is a probability distribution, representing uncertainty about $\theta$ after seeing the dataset. Bayesian inference is a prime example of such a procedure but one can also construct other learning procedures that return distributional output. This paper studies conditions for a learning procedure to be considered calibrated, in the sense that the true data-generating parameters are plausible as samples from its distributional output. A learning procedure that is calibrated need not be statistically efficient and vice versa. A hypothesis-testing framework is developed in order to assess, using simulation, whether a learning procedure is calibrated. Finally, we exploit our framework to test the calibration of some learning procedures that are motivated as being approximations to Bayesian inference but are nevertheless widely used

Testing whether a learning procedure is calibrated

A learning procedure takes as input a dataset and performs inference for the parameters θ of a model that is assumed to have given rise to the dataset. Here we consider learning procedures whose output is a probability distribution, representing uncertainty about θ after seeing the dataset. Bayesian inference is a prime example of such a procedure, but one can also construct other learning procedures that return distributional output. This paper studies conditions for a learning procedure to be considered calibrated, in the sense that the true data-generating parameters are plausible as samples from its distributional output. A learning procedure whose inferences and predictions are systematically over- or under-confident will fail to be calibrated. On the other hand, a learning procedure that is calibrated need not be statistically efficient. A hypothesis-testing framework is developed in order to assess, using simulation, whether a learning procedure is calibrated. Several vignettes are presented to illustrate different aspects of the framework

The impact of migration on tuberculosis epidemiology and control in high-income countries: a review.

Tuberculosis (TB) causes significant morbidity and mortality in high-income countries with foreign-born individuals bearing a disproportionate burden of the overall TB case burden in these countries. In this review of tuberculosis and migration we discuss the impact of migration on the epidemiology of TB in low burden countries, describe the various screening strategies to address this issue, review the yield and cost-effectiveness of these programs and describe the gaps in knowledge as well as possible future solutions.The reasons for the TB burden in the migrant population are likely to be the reactivation of remotely-acquired latent tuberculosis infection (LTBI) following migration from low/intermediate-income high TB burden settings to high-income, low TB burden countries.TB control in high-income countries has historically focused on the early identification and treatment of active TB with accompanying contact-tracing. In the face of the TB case-load in migrant populations, however, there is ongoing discussion about how best to identify TB in migrant populations. In general, countries have generally focused on two methods: identification of active TB (either at/post-arrival or increasingly pre-arrival in countries of origin) and secondly, conditionally supported by WHO guidance, through identifying LTBI in migrants from high TB burden countries. Although health-economic analyses have shown that TB control in high income settings would benefit from providing targeted LTBI screening and treatment to certain migrants from high TB burden countries, implementation issues and barriers such as sub-optimal treatment completion will need to be addressed to ensure program efficacy

A Bayesian nonparametric test for conditional independence

This article introduces a Bayesian nonparametric method for quantifying the relative evidence in a dataset in favour of the dependence or independence of two variables conditional on a third. The approach uses Pólya tree priors on spaces of conditional probability densities, accounting for uncertainty in the form of the underlying distributions in a nonparametric way. The Bayesian perspective provides an inherently symmetric probability measure of conditional dependence or independence, a feature particularly advantageous in causal discovery and not employed in existing procedures of this type

Contributed Discussion on Article by Chkrebtii, Campbell, Calderhead, and Girolami

We commend the authors for an exciting paper which provides a strong contribution to the emerging field of probabilistic numerics (PN). Below, we discuss aspects of prior modelling which need to be considered thoroughly in future wor

Implicit probabilistic integrators for ODEs

We introduce a family of implicit probabilistic integrators for initial value problems (IVPs), taking as a starting point the multistep Adams–Moulton method. The implicit construction allows for dynamic feedback from the forthcoming time-step, in contrast to previous probabilistic integrators, all of which are based on explicit methods. We begin with a concise survey of the rapidly-expanding field of probabilistic ODE solvers. We then introduce our method, which builds on and adapts the work of Conrad et al. (2016) and Teymur et al. (2016), and provide a rigorous proof of its well-definedness and convergence. We discuss the problem of the calibration of such integrators and suggest one approach. We give an illustrative example highlighting the effect of the use of probabilistic integrators—including our new method—in the setting of parameter inference within an inverse problem