Model choice versus model criticism

The new perspectives on ABC and Bayesian model criticisms presented in Ratmann et al.(2009) are challenging standard approaches to Bayesian model choice. We discuss here some issues arising from the authors' approach, including prior influence, model assessment and criticism, and the meaning of error in ABC.Comment: This is a comment on the recent paper by Ratmann, Andrieu, Wiuf, and Richardson (PNAS, 106), submitted too late for PNAS to consider i

Reliable ABC model choice via random forests

Approximate Bayesian computation (ABC) methods provide an elaborate approach to Bayesian inference on complex models, including model choice. Both theoretical arguments and simulation experiments indicate, however, that model posterior probabilities may be poorly evaluated by standard ABC techniques. We propose a novel approach based on a machine learning tool named random forests to conduct selection among the highly complex models covered by ABC algorithms. We thus modify the way Bayesian model selection is both understood and operated, in that we rephrase the inferential goal as a classification problem, first predicting the model that best fits the data with random forests and postponing the approximation of the posterior probability of the predicted MAP for a second stage also relying on random forests. Compared with earlier implementations of ABC model choice, the ABC random forest approach offers several potential improvements: (i) it often has a larger discriminative power among the competing models, (ii) it is more robust against the number and choice of statistics summarizing the data, (iii) the computing effort is drastically reduced (with a gain in computation efficiency of at least fifty), and (iv) it includes an approximation of the posterior probability of the selected model. The call to random forests will undoubtedly extend the range of size of datasets and complexity of models that ABC can handle. We illustrate the power of this novel methodology by analyzing controlled experiments as well as genuine population genetics datasets. The proposed methodologies are implemented in the R package abcrf available on the CRAN.Comment: 39 pages, 15 figures, 6 table

Notes to Robert et al.: Model criticism informs model choice and model comparison

In their letter to PNAS and a comprehensive set of notes on arXiv [arXiv:0909.5673v2], Christian Robert, Kerrie Mengersen and Carla Chen (RMC) represent our approach to model criticism in situations when the likelihood cannot be computed as a way to "contrast several models with each other". In addition, RMC argue that model assessment with Approximate Bayesian Computation under model uncertainty (ABCmu) is unduly challenging and question its Bayesian foundations. We disagree, and clarify that ABCmu is a probabilistically sound and powerful too for criticizing a model against aspects of the observed data, and discuss further the utility of ABCmu.Comment: Reply to [arXiv:0909.5673v2

Bayesian Model Choice of Grouped t-copula

One of the most popular copulas for modeling dependence structures is t-copula. Recently the grouped t-copula was generalized to allow each group to have one member only, so that a priori grouping is not required and the dependence modeling is more flexible. This paper describes a Markov chain Monte Carlo (MCMC) method under the Bayesian inference framework for estimating and choosing t-copula models. Using historical data of foreign exchange (FX) rates as a case study, we found that Bayesian model choice criteria overwhelmingly favor the generalized t-copula. In addition, all the criteria also agree on the second most likely model and these inferences are all consistent with classical likelihood ratio tests. Finally, we demonstrate the impact of model choice on the conditional Value-at-Risk for portfolios of six major FX rates

An Agent-based Route Choice Model

Travel demand emerges from individual decisions. These decisions, depending on individual objectives, preferences, experiences and spatial knowledge about travel, are both heterogeneous and evolutionary. Research emerging from fields such as road pricing and ATIS requires travel demand models that are able to consider travelers with distinct attributes (value of time (VOT), willingness to pay, travel budgets, etc.) and behavioral preferences (e.g. willingness to switch routes with potential savings) in a differentiated market (by tolls and the level of service). Traditional trip-based models have difficulty in dealing with the aforementioned heterogeneity and issues such as equity. Moreover, the role of spatial information, which has significant influence on decision-making and travel behavior, has not been fully addressed in existing models. To bridge the gap, this paper proposes to explicitly model the formation and spread- ing of spatial knowledge among travelers. An Agent-based Route Choice (ARC) model was developed to track choices of each decision-maker on a road network over time and map individual choices into macroscopic flow pattern. ARC has been applied on both SiouxFalls network and Chicago sketch network. Comparison between ARC and existing models (UE and SUE) on both networks shows ARC is valid and computationally tractable. To be brief, this paper specifically focuses on the route choice behavior, while the proposed model can be extended to other modules of travel demand under an integrated framework.Agent-based model, route choice, traffic assignment, travel demand modeling

Delving into Choice Internals: A Joint Discrete Choice/Attribute Rating Model

Multi-attribute modeling has rapidly progressed from being a novelty to becoming a widely used tool of economic research. When both choice and attribute ratings data are available, a model that makes joint use of both offers informative inference opportunities. In the present study we develop a joint model which utilizes both choice and ratings data, allows for scale usage heterogeneity, is robust to violations of utility continuity and completeness. The model is used to obtain WTP estimates for genetically-modified content and country-of-origin attributes in an survey-based study of Canola oil labeling. The median survey respondent's WTP for non-GM Canola oil was found to be CA$0.92/liter. The median WTP for non-specification of GM content was found to be approximately 80$0.86/liter.Demand and Price Analysis, Institutional and Behavioral Economics,