63 research outputs found
Some models are useful, but how do we know which ones? Towards a unified Bayesian model taxonomy
Probabilistic (Bayesian) modeling has experienced a surge of applications in
almost all quantitative sciences and industrial areas. This development is
driven by a combination of several factors, including better probabilistic
estimation algorithms, flexible software, increased computing power, and a
growing awareness of the benefits of probabilistic learning. However, a
principled Bayesian model building workflow is far from complete and many
challenges remain. To aid future research and applications of a principled
Bayesian workflow, we ask and provide answers for what we perceive as two
fundamental questions of Bayesian modeling, namely (a) "What actually is a
Bayesian model?" and (b) "What makes a good Bayesian model?". As an answer to
the first question, we propose the PAD model taxonomy that defines four basic
kinds of Bayesian models, each representing some combination of the assumed
joint distribution of all (known or unknown) variables (P), a posterior
approximator (A), and training data (D). As an answer to the second question,
we propose ten utility dimensions according to which we can evaluate Bayesian
models holistically, namely, (1) causal consistency, (2) parameter
recoverability, (3) predictive performance, (4) fairness, (5) structural
faithfulness, (6) parsimony, (7) interpretability, (8) convergence, (9)
estimation speed, and (10) robustness. Further, we propose two example utility
decision trees that describe hierarchies and trade-offs between utilities
depending on the inferential goals that drive model building and testing
Fuse It or Lose It: Deep Fusion for Multimodal Simulation-Based Inference
We present multimodal neural posterior estimation (MultiNPE), a method to
integrate heterogeneous data from different sources in simulation-based
inference with neural networks. Inspired by advances in deep fusion learning,
it empowers researchers to analyze data from different domains and infer the
parameters of complex mathematical models with increased accuracy. We formulate
multimodal fusion approaches for \hbox{MultiNPE} (early, late, hybrid) and
evaluate their performance in three challenging experiments. MultiNPE not only
outperforms single-source baselines on a reference task, but also achieves
superior inference on scientific models from neuroscience and cardiology. We
systematically investigate the impact of partially missing data on the
different fusion strategies. Across our experiments, late and hybrid fusion
techniques emerge as the methods of choice for practical applications of
multimodal simulation-based inference
Validation and Comparison of Non-Stationary Cognitive Models: A Diffusion Model Application
Cognitive processes undergo various fluctuations and transient states across
different temporal scales. Superstatistics are emerging as a flexible framework
for incorporating such non-stationary dynamics into existing cognitive model
classes. In this work, we provide the first experimental validation of
superstatistics and formal comparison of four non-stationary diffusion decision
models in a specifically designed perceptual decision-making task. Task
difficulty and speed-accuracy trade-off were systematically manipulated to
induce expected changes in model parameters. To validate our models, we assess
whether the inferred parameter trajectories align with the patterns and
sequences of the experimental manipulations. To address computational
challenges, we present novel deep learning techniques for amortized Bayesian
estimation and comparison of models with time-varying parameters. Our findings
indicate that transition models incorporating both gradual and abrupt parameter
shifts provide the best fit to the empirical data. Moreover, we find that the
inferred parameter trajectories closely mirror the sequence of experimental
manipulations. Posterior re-simulations further underscore the ability of the
models to faithfully reproduce critical data patterns. Accordingly, our results
suggest that the inferred non-stationary dynamics may reflect actual changes in
the targeted psychological constructs. We argue that our initial experimental
validation paves the way for the widespread application of superstatistics in
cognitive modeling and beyond
A Deep Learning Method for Comparing Bayesian Hierarchical Models
Bayesian model comparison (BMC) offers a principled approach for assessing
the relative merits of competing computational models and propagating
uncertainty into model selection decisions. However, BMC is often intractable
for the popular class of hierarchical models due to their high-dimensional
nested parameter structure. To address this intractability, we propose a deep
learning method for performing BMC on any set of hierarchical models which can
be instantiated as probabilistic programs. Since our method enables amortized
inference, it allows efficient re-estimation of posterior model probabilities
and fast performance validation prior to any real-data application. In a series
of extensive validation studies, we benchmark the performance of our method
against the state-of-the-art bridge sampling method and demonstrate excellent
amortized inference across all BMC settings. We then showcase our method by
comparing four hierarchical evidence accumulation models that have previously
been deemed intractable for BMC due to partly implicit likelihoods. In this
application, we corroborate evidence for the recently proposed L\'evy flight
model of decision-making and show how transfer learning can be leveraged to
enhance training efficiency. We provide reproducible code for all analyses and
an open-source implementation of our method
Towards Context-Aware Domain Generalization: Understanding the Benefits and Limits of Marginal Transfer Learning
In this work, we analyze the conditions under which information about the
context of an input can improve the predictions of deep learning models in
new domains. Following work in marginal transfer learning in Domain
Generalization (DG), we formalize the notion of context as a
permutation-invariant representation of a set of data points that originate
from the same domain as the input itself. We offer a theoretical analysis of
the conditions under which this approach can, in principle, yield benefits, and
formulate two necessary criteria that can be easily verified in practice.
Additionally, we contribute insights into the kind of distribution shifts for
which the marginal transfer learning approach promises robustness. Empirical
analysis shows that our criteria are effective in discerning both favorable and
unfavorable scenarios. Finally, we demonstrate that we can reliably detect
scenarios where a model is tasked with unwarranted extrapolation in
out-of-distribution (OOD) domains, identifying potential failure cases.
Consequently, we showcase a method to select between the most predictive and
the most robust model, circumventing the well-known trade-off between
predictive performance and robustness
Neural Superstatistics for Bayesian Estimation of Dynamic Cognitive Model
Mathematical models of cognition are often memoryless and ignore potential
fluctuations of their parameters. However, human cognition is inherently
dynamic. Thus, we propose to augment mechanistic cognitive models with a
temporal dimension and estimate the resulting dynamics from a superstatistics
perspective. Such a model entails a hierarchy between a low-level observation
model and a high-level transition model. The observation model describes the
local behavior of a system, and the transition model specifies how the
parameters of the observation model evolve over time. To overcome the
estimation challenges resulting from the complexity of superstatistical models,
we develop and validate a simulation-based deep learning method for Bayesian
inference, which can recover both time-varying and time-invariant parameters.
We first benchmark our method against two existing frameworks capable of
estimating time-varying parameters. We then apply our method to fit a dynamic
version of the diffusion decision model to long time series of human response
times data. Our results show that the deep learning approach is very efficient
in capturing the temporal dynamics of the model. Furthermore, we show that the
erroneous assumption of static or homogeneous parameters will hide important
temporal information
- …