Model Assessment Tools for a Model False World

A standard goal of model evaluation and selection is to find a model that approximates the truth well while at the same time is as parsimonious as possible. In this paper we emphasize the point of view that the models under consideration are almost always false, if viewed realistically, and so we should analyze model adequacy from that point of view. We investigate this issue in large samples by looking at a model credibility index, which is designed to serve as a one-number summary measure of model adequacy. We define the index to be the maximum sample size at which samples from the model and those from the true data generating mechanism are nearly indistinguishable. We use standard notions from hypothesis testing to make this definition precise. We use data subsampling to estimate the index. We show that the definition leads us to some new ways of viewing models as flawed but useful. The concept is an extension of the work of Davies [Statist. Neerlandica 49 (1995) 185--245].Comment: Published in at http://dx.doi.org/10.1214/09-STS302 the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org

Modeling good research practices - overview: a report of the ISPOR-SMDM modeling good research practices task force - 1.

Models—mathematical frameworks that facilitate estimation of the consequences of health care decisions—have become essential tools for health technology assessment. Evolution of the methods since the first ISPOR modeling task force reported in 2003 has led to a new task force, jointly convened with the Society for Medical Decision Making, and this series of seven papers presents the updated recommendations for best practices in conceptualizing models; implementing state–transition approaches, discrete event simulations, or dynamic transmission models; dealing with uncertainty; and validating and reporting models transparently. This overview introduces the work of the task force, provides all the recommendations, and discusses some quandaries that require further elucidation. The audience for these papers includes those who build models, stakeholders who utilize their results, and, indeed, anyone concerned with the use of models to support decision making

Simulated Clinical Trias: some design issues

Simulation is widely used to investigate real-world systems in a large number of fields, including clinical trials for drug development, since real trials are costly, frequently fail and may lead to serious side effects. This paper is a survey of the statistical issues arising in these simulated trials. We illustrate the broad applicability of this investigation tool by means of examples selected from the literature. We discuss the aims and the peculiarities of the simulation models used in this context, including a brief mention of the use of metamodels. Of special interest is the topic of the design of the virtual experiments, stressing similarities and differences with the design of real life trials. Since it is important for a computerized model to possess a satisfactory range of accuracy consistent with its intended application, real data provided by physical experiments are used to confirm the simulator : we illustrate validating techniques through a number of examples. We end the paper with some challenging questions on the scientificity, ethics and effectiveness of simulation in the clinical research, and the interesting research problem of how to integrate simulated and physical experiments in a clinical context.Simulation models; pharmacokinetics; pharmacodynamics; model validation; experimental design, ethics. Modelli di simulazione; farmacocinetica; farmacodinamica; validazione; disegno degli esperimenti; etica.

Overview on agent-based social modelling and the use of formal languages

Transdisciplinary Models and Applications investigates a variety of programming languages used in validating and verifying models in order to assist in their eventual implementation. This book will explore different methods of evaluating and formalizing simulation models, enabling computer and industrial engineers, mathematicians, and students working with computer simulations to thoroughly understand the progression from simulation to product, improving the overall effectiveness of modeling systems.Postprint (author's final draft

Reviews

Danny Saunders and Nina Smalley (eds.), The International Simulation and Gaming Research Yearbook — Volume 8: Simulations and Games for Transition and Change, London: Kogan Page, 2000. ISBN: 0–7494–3397–3. Hardback, viii+271 pages, £40.00

Traveller Behaviour: Decision making in an unpredictable world

This paper discusses the nature and consequences of uncertainty in transport systems. Drawing on work from a number of fields, it addresses travellers’ abilities to predict variable phenomena, their perception of uncertainty, their attitude to risk and the various strategies they might adopt in response to uncertainty. It is argued that despite the increased interest in the representation of uncertainty in transport systems, most models treat uncertainty as a purely statistical issue and ignore the psychological aspects of response to uncertainty. The principle theories and models currently used to predict travellers’ response to uncertainty are presented and number of alternative modelling approaches are outlined. It is argued that the current generation of predictive models do not provide an adequate basis for forecasting response to changes in the degree of uncertainty or for predicting the likely effect of providing additional information. A number of alternative modelling approaches are identified to deal with travellers’ acquisition of information, the definition of their choice set and their choice between the available options. The use of heuristic approaches is recommended as an alternative to more conventional probabilistic methods

A framework for characterising and evaluating the effectiveness of environmental modelling

Environmental modelling is transitioning from the traditional paradigm that focuses on the model and its quantitative performance to a more holistic paradigm that recognises successful model-based outcomes are closely tied to undertaking modelling as a social process, not just as a technical procedure. This paper redefines evaluation as a multi-dimensional and multi-perspective concept, and proposes a more complete framework for identifying and measuring the effectiveness of modelling that serves the new paradigm. Under this framework, evaluation considers a broader set of success criteria, and emphasises the importance of contextual factors in determining the relevance and outcome of the criteria. These evaluation criteria are grouped into eight categories: project efficiency, model accessibility, credibility, saliency, legitimacy, satisfaction, application, and impact. Evaluation should be part of an iterative and adaptive process that attempts to improve model-based outcomes and foster pathways to better futures