Holistically Evaluating Agent Based Social System Models

The philosophical perspectives on model evaluation can be broadly classified into reductionist/logical positivist and relativist/holistic. In this paper, we outline some of our past efforts in, and challenges faced during, evaluating models of social systems with cognitively detailed agents. Owing to richness in the model, we argue that the holistic approach and consequent continuous improvement are essential to evaluating complex social system models such as these. A social system built primarily of cognitively detailed agents can provide multiple levels of correspondence, both at observable and abstract aggregated levels. Such a system can also pose several challenges, including large feature spaces, issues in information elicitation with database, experts and news feeds, counterfactuals, fragmented theoretical base, and limited funding for validation. We subscribe to the view that no model can faithfully represent reality, but detailed, descriptive models are useful in learning about the system and bringing about a qualitative jump in understanding of the system it attempts to model – provided they are properly validated. Our own approach to model evaluation is to consider the entire life cycle and assess the validity under two broad dimensions of (1) internally focused validity/quality achieved through structural, methodological, and ontological evaluations; and (2) external validity consisting of micro validity, macro validity, and qualitative, causal and narrative validity. In this paper, we also elaborate on selected validation techniques that we have employed in the past. We recommend a triangulation of multiple validation techniques, including methodological soundness, qualitative validation techniques, such as face validation by experts and narrative validation, and formal validation tests, including correspondence testing

Reusing simulation experiments for model composition and extension

This thesis aims to reuse simulation experiments to support developing models via model reuse, with a focus on validating the resulting model. Individual models are annotated with their simulation experiments. Upon reuse of those models for building new ones, the associated simulation experiments are also reused and executed with the new model, to inspect whether the key behavior exhibited by the original models is preserved or not in the new model. Hence, the changes of model behavior resulting from the model reuse are revealed, and insights into validity of the new model are provided

Toward composing variable structure models and their interfaces: a case of intensional coupling definitions

In this thesis, we investigate a combination of traditional component-based and variable structure modeling. The focus is on a structural consistent specification of couplings in modular, hierarchical models with a variable structure. For this, we exploitintensional definitions, as known from logic, and introduce a novel intensional coupling definition, which allows a concise yet expressive specification of complex communication and interaction patterns in static as well as variable structure models, without the need to worryabout structural consistency.In der Arbeit untersuchen wir ein Zusammenbringen von klassischer komponenten-basierter und variabler Strukturmodellierung. Der Fokus liegt dabei auf der Spezifikation von strukturkonsistenten Kopplungen in modular-hierarchischen Modellen mit einer variablen Struktur. Dafür nutzen wir intensionale Definitionen, wie sie aus der Logik bekannt sind, und führen ein neuartiges Konzept von intensionalen Kopplungen ein, welches kompakte gleichzeitig ausdrucksstarke Spezifikationen von komplexen Kommunikations- und Interaktionsmuster in statischen und variablen Strukturmodellen erlaubt

An approach for validation of semantic composability in simulation models

10.1109/PADS.2009.14Proceedings - Workshop on Principles of Advanced and Distributed Simulation, PADS3-1