EEMCS final report for the causal modeling for air transport safety (CATS) project

This document reports on the work realized by the DIAM in relation to the completion of the CATS model as presented in Figure 1.6 and tries to explain some of the steps taken for its completion. The project spans over a period of time of three years. Intermediate reports have been presented throughout the project’s progress. These are presented in Appendix 1. In this report the continuous‐discrete distribution‐free BBNs are briefly discussed. The human reliability models developed for dealing with dependence in the model variables are described and the software application UniNet is presente

Expert Elicitation for Reliable System Design

This paper reviews the role of expert judgement to support reliability assessments within the systems engineering design process. Generic design processes are described to give the context and a discussion is given about the nature of the reliability assessments required in the different systems engineering phases. It is argued that, as far as meeting reliability requirements is concerned, the whole design process is more akin to a statistical control process than to a straightforward statistical problem of assessing an unknown distribution. This leads to features of the expert judgement problem in the design context which are substantially different from those seen, for example, in risk assessment. In particular, the role of experts in problem structuring and in developing failure mitigation options is much more prominent, and there is a need to take into account the reliability potential for future mitigation measures downstream in the system life cycle. An overview is given of the stakeholders typically involved in large scale systems engineering design projects, and this is used to argue the need for methods that expose potential judgemental biases in order to generate analyses that can be said to provide rational consensus about uncertainties. Finally, a number of key points are developed with the aim of moving toward a framework that provides a holistic method for tracking reliability assessment through the design process.Comment: This paper commented in: [arXiv:0708.0285], [arXiv:0708.0287], [arXiv:0708.0288]. Rejoinder in [arXiv:0708.0293]. Published at http://dx.doi.org/10.1214/088342306000000510 in the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org

Ordered Preference Elicitation Strategies for Supporting Multi-Objective Decision Making

In multi-objective decision planning and learning, much attention is paid to producing optimal solution sets that contain an optimal policy for every possible user preference profile. We argue that the step that follows, i.e, determining which policy to execute by maximising the user's intrinsic utility function over this (possibly infinite) set, is under-studied. This paper aims to fill this gap. We build on previous work on Gaussian processes and pairwise comparisons for preference modelling, extend it to the multi-objective decision support scenario, and propose new ordered preference elicitation strategies based on ranking and clustering. Our main contribution is an in-depth evaluation of these strategies using computer and human-based experiments. We show that our proposed elicitation strategies outperform the currently used pairwise methods, and found that users prefer ranking most. Our experiments further show that utilising monotonicity information in GPs by using a linear prior mean at the start and virtual comparisons to the nadir and ideal points, increases performance. We demonstrate our decision support framework in a real-world study on traffic regulation, conducted with the city of Amsterdam.Comment: AAMAS 2018, Source code at https://github.com/lmzintgraf/gp_pref_elici

A literature review on the use of expert opinion in probabilistic risk analysis

Risk assessment is part of the decision making process in many fields of discipline, such as engineering, public health, environment, program management, regulatory policy, and finance. There has been considerable debate over the philosophical and methodological treatment of risk in the past few decades, ranging from its definition and classification to methods of its assessment. Probabilistic risk analysis (PRA) specifically deals with events represented by low probabilities of occurring with high levels of unfavorable consequences. Expert judgment is often a critical source of information in PRA, since empirical data on the variables of interest are rarely available. The author reviews the literature on the use of expert opinion in PRA, in particular on the approaches to eliciting and aggregating experts'assessments. The literature suggests that the methods by which expert opinions are collected and combined have a significant effect on the resulting estimates. The author discusses two types of approaches to eliciting and aggregating expert judgments-behavioral and mathematical approaches, with the emphasis on the latter. It is generally agreed that mathematical approaches tend to yield more accurate estimates than behavioral approaches. After a short description of behavioral approaches, the author discusses mathematical approaches in detail, presenting three aggregation models: non-Bayesian axiomatic models, Bayesian models, andpsychological scaling models. She also discusses issues of stochastic dependence.Health Monitoring&Evaluation,ICT Policy and Strategies,Public Health Promotion,Enterprise Development&Reform,Statistical&Mathematical Sciences,ICT Policy and Strategies,Health Monitoring&Evaluation,Statistical&Mathematical Sciences,Science Education,Scientific Research&Science Parks

Comparing stochastic design decision belief models : pointwise versus interval probabilities.

Decision support systems can either directly support a product designer or support an agent operating within a multi-agent system (MAS). Stochastic based decision support systems require an underlying belief model that encodes domain knowledge. The underlying supporting belief model has traditionally been a probability distribution function (PDF) which uses pointwise probabilities for all possible outcomes. This can present a challenge during the knowledge elicitation process. To overcome this, it is proposed to test the performance of a credal set belief model. Credal sets (sometimes also referred to as p-boxes) use interval probabilities rather than pointwise probabilities and therefore are more easier to elicit from domain experts. The PDF and credal set belief models are compared using a design domain MAS which is able to learn, and thereby refine, the belief model based on its experience. The outcome of the experiment illustrates that there is no significant difference between the PDF based and credal set based belief models in the performance of the MAS