3 research outputs found
Sequential Bayesian Experimental Design for Implicit Models via Mutual Information
Bayesian experimental design (BED) is a framework that uses statistical
models and decision making under uncertainty to optimise the cost and
performance of a scientific experiment. Sequential BED, as opposed to static
BED, considers the scenario where we can sequentially update our beliefs about
the model parameters through data gathered in the experiment. A class of models
of particular interest for the natural and medical sciences are implicit
models, where the data generating distribution is intractable, but sampling
from it is possible. Even though there has been a lot of work on static BED for
implicit models in the past few years, the notoriously difficult problem of
sequential BED for implicit models has barely been touched upon. We address
this gap in the literature by devising a novel sequential design framework for
parameter estimation that uses the Mutual Information (MI) between model
parameters and simulated data as a utility function to find optimal
experimental designs, which has not been done before for implicit models. Our
approach uses likelihood-free inference by ratio estimation to simultaneously
estimate posterior distributions and the MI. During the sequential BED
procedure we utilise Bayesian optimisation to help us optimise the MI utility.
We find that our framework is efficient for the various implicit models tested,
yielding accurate parameter estimates after only a few iterations
Statistical applications of contrastive learning
The likelihood function plays a crucial role in statistical inference and
experimental design. However, it is computationally intractable for several
important classes of statistical models, including energy-based models and
simulator-based models. Contrastive learning is an intuitive and
computationally feasible alternative to likelihood-based learning. We here
first provide an introduction to contrastive learning and then show how we can
use it to derive methods for diverse statistical problems, namely parameter
estimation for energy-based models, Bayesian inference for simulator-based
models, as well as experimental design.Comment: Accepted to Behaviormetrik
Designing Optimal Behavioral Experiments Using Machine Learning
Computational models are powerful tools for understanding human cognition and behavior. They let us express our theories clearly and precisely, and offer predictions that can be subtle and often counter-intuitive. However, this same richness and ability to surprise means our scientific intuitions and traditional tools are ill-suited to designing experiments to test and compare these models. To avoid these pitfalls and realize the full potential of computational modeling, we require tools to design experiments that provide clear answers about what models explain human behavior and the auxiliary assumptions those models must make. Bayesian optimal experimental design (BOED) formalizes the search for optimal experimental designs by identifying experiments that are expected to yield informative data. In this work, we provide a tutorial on leveraging recent advances in BOED and machine learning to find optimal experiments for any kind of model that we can simulate data from, and show how by-products of this procedure allow for quick and straightforward evaluation of models and their parameters against real experimental data. As a case study, we consider theories of how people balance exploration and exploitation in multi-armed bandit decision-making tasks. We validate the presented approach using simulations and a real-world experiment. As compared to experimental designs commonly used in the literature, we show that our optimal designs more efficiently determine which of a set of models best account for individual human behavior, and more efficiently characterize behavior given a preferred model. At the same time, formalizing a scientific question such that it can be adequately addressed with BOED can be challenging and we discuss several potential caveats and pitfalls that practitioners should be aware of. We provide code and tutorial notebooks to replicate all analyses