Uncertainty Assessment in High-Risk Environments Using Probability, Evidence Theory and Expert Judgment Elicitation

The level of uncertainty in advanced system design is assessed by comparing the results of expert judgment elicitation to probability and evidence theory. This research shows how one type of monotone measure, namely Dempster-Shafer Theory of Evidence can expand the framework of uncertainty to provide decision makers a more robust solution space. The issues imbedded in this research are focused on how the relevant predictive uncertainty produced by similar action is measured. This methodology uses the established approach from traditional probability theory and Dempster-Shafer evidence theory to combine two classes of uncertainty, aleatory and epistemic. Probability theory provides the mathematical structure traditionally used in the representation of aleatory uncertainty. The uncertainty in analysis outcomes is represented by probability distributions and typically summarized as Complimentary Cumulative Distribution Functions (CCDFs). The main components of this research are probability of X in the probability theory compared to mx in evidence theory. Using this comparison, an epistemic model is developed to obtain the upper “CCPF - Complimentary Cumulative Plausibility Function” limits and the lower “CCBF - Complimentary Cumulative Belief Function” limits compared to the traditional probability function. A conceptual design for the Thermal Protection System (TPS) of future Crew Exploration Vehicles (CEV) is used as an initial test case. A questionnaire is tailored to elicit judgment from experts in high-risk environments. Based on description and characteristics, the answers of the questionnaire produces information, that serves as qualitative semantics used for the evidence theory functions. The computational mechanism provides a heuristic approach for the compilation and presentation of the results. A follow-up evaluation serves as validation of the findings and provides useful information in terms of consistency and adoptability to other domains. The results of this methodology provide a useful and practical approach in conceptual design to aid the decision maker in assessing the level of uncertainty of the experts. The methodology presented is well-suited for decision makers that encompass similar conceptual design instruments

Prior knowledge elicitation: The past, present, and future

Specification of the prior distribution for a Bayesian model is a central part of the Bayesian workflow for data analysis, but it is often difficult even for statistical experts. Prior elicitation transforms domain knowledge of various kinds into well-defined prior distributions, and offers a solution to the prior specification problem, in principle. In practice, however, we are still fairly far from having usable prior elicitation tools that could significantly influence the way we build probabilistic models in academia and industry. We lack elicitation methods that integrate well into the Bayesian workflow and perform elicitation efficiently in terms of costs of time and effort. We even lack a comprehensive theoretical framework for understanding different facets of the prior elicitation problem.Why are we not widely using prior elicitation? We analyze the state of the art by identifying a range of key aspects of prior knowledge elicitation, from properties of the modelling task and the nature of the priors to the form of interaction with the expert. The existing prior elicitation literature is reviewed and categorized in these terms. This allows recognizing under-studied directions in prior elicitation research, finally leading to a proposal of several new avenues to improve prior elicitation methodology.Fil: Mikkola, Petrus. Aalto University; FinlandiaFil: Martín, Osvaldo Antonio. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - San Luis. Instituto de Matemática Aplicada de San Luis "Prof. Ezio Marchi". Universidad Nacional de San Luis. Facultad de Ciencias Físico, Matemáticas y Naturales. Instituto de Matemática Aplicada de San Luis "Prof. Ezio Marchi"; Argentina. Aalto University; FinlandiaFil: Chandramoul, Suyog. Aalto University; FinlandiaFil: Hartmann, Marcelo. University of Helsinki; FinlandiaFil: Abril Pla, Oriol. University of Helsinki; FinlandiaFil: Thomas, Owen. University of Oslo; NoruegaFil: Pesonen, Henri. University of Oslo; NoruegaFil: Corander, Jukka. University of Oslo; NoruegaFil: Vehtari, Aki. Aalto University; FinlandiaFil: Kaski, Samuel. Aalto University; FinlandiaFil: Bürkner, Paul Christian. University Of Stuttgart; AlemaniaFil: Klami, Arto. University of Helsinki; Finlandi

Prior knowledge elicitation: The past, present, and future

Specification of the prior distribution for a Bayesian model is a central part of the Bayesian workflow for data analysis, but it is often difficult even for statistical experts. Prior elicitation transforms domain knowledge of various kinds into well-defined prior distributions, and offers a solution to the prior specification problem, in principle. In practice, however, we are still fairly far from having usable prior elicitation tools that could significantly influence the way we build probabilistic models in academia and industry. We lack elicitation methods that integrate well into the Bayesian workflow and perform elicitation efficiently in terms of costs of time and effort. We even lack a comprehensive theoretical framework for understanding different facets of the prior elicitation problem.Why are we not widely using prior elicitation? We analyze the state of the art by identifying a range of key aspects of prior knowledge elicitation, from properties of the modelling task and the nature of the priors to the form of interaction with the expert. The existing prior elicitation literature is reviewed and categorized in these terms. This allows recognizing under-studied directions in prior elicitation research, finally leading to a proposal of several new avenues to improve prior elicitation methodology.Fil: Mikkola, Petrus. Aalto University; FinlandiaFil: Martín, Osvaldo Antonio. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - San Luis. Instituto de Matemática Aplicada de San Luis "Prof. Ezio Marchi". Universidad Nacional de San Luis. Facultad de Ciencias Físico, Matemáticas y Naturales. Instituto de Matemática Aplicada de San Luis "Prof. Ezio Marchi"; Argentina. Aalto University; FinlandiaFil: Chandramoul, Suyog. Aalto University; FinlandiaFil: Hartmann, Marcelo. University of Helsinki; FinlandiaFil: Abril Pla, Oriol. University of Helsinki; FinlandiaFil: Thomas, Owen. University of Oslo; NoruegaFil: Pesonen, Henri. University of Oslo; NoruegaFil: Corander, Jukka. University of Oslo; NoruegaFil: Vehtari, Aki. Aalto University; FinlandiaFil: Kaski, Samuel. Aalto University; FinlandiaFil: Bürkner, Paul Christian. University Of Stuttgart; AlemaniaFil: Klami, Arto. University of Helsinki; Finlandi

Assimilating Non-Probabilistic Assessments of the Estimation of Uncertainty Bias in Expert Judgment Elicitation Using an Evidence Based Approach in High Consequence Conceptual Designs

One of the major challenges in conceptual designs of complex systems is the identification of uncertainty embedded in the information due to lack of historic data. This becomes of increased concern especially in high-risk industries. This document reports a developed methodology that allows for the cognitive bias, estimation of uncertainty, to be elucidated to improve the quality of elicited data. It consists of a comprehensive literature review that begins by defining a \u27High Consequence Conceptual Engineering Environment\u27 and identifies the high-risk industries in which these environments are found. It proceeds with a discussion that differentiates risk and uncertainty in decision-making in these environments. An argument was built around the identified epistemic category of uncertainty, the impact on hard data for decision-making, and from whom we obtain this data. The review shifts to defining and selecting the experts, the elicitation process in terms of the components, the process phases and steps involved, and an examination of a probabilistic and a fuzzy example. This sets the stage for this methodology that uses evidence theory for the mathematical analysis after the data is elicited using a tailored elicitation process. Yager\u27s combination rule is used to combine evidence and fully recognize the ignorance without ignoring available information. Engineering and management teams from NASA Langley Research Center were the population from which the experts for this study were identified. NASA officials were interested in obtaining uncertainty estimates, and a comparison of these estimates, associated with their Crew Launch Vehicle (CLV) designs; the existing Exploration Systems Architecture Study Crew Launch Vehicle (ESAS CLV) and the Parallel-Staged Crew Launch Vehicle (P-S CLV) which is currently being worked. This evidence-based approach identified that the estimation of cost parameters uncertainty is not specifically over or underestimated in High Consequence Conceptual Engineering Environments; rather, there is more uncertainty present than what is being anticipated. From the perspective of maturing designs, it was concluded that the range of cost parameters\u27 uncertainty at different error-state-values were interchangeably larger or smaller when compared to each other even as the design matures

An early-stage decision-support framework for the implementation of intelligent automation

The constant pressure on manufacturing companies to improve productivity, reduce the lead time and progress in quality requires new technological developments and adoption.The rapid development of smart technology and robotics and autonomous systems (RAS) technology has a profound impact on manufacturing automation and might determine winners and losers of the next generation’s manufacturing competition. Simultaneously, recent smart technology developments in the areas enable an automation response to new production paradigms such as mass customisation and product-lifecycle considerations in the context of Industry 4.0. New paradigms, like mass customisation, increased both the complexity of the tasks and the risk due to smart technology integration. From a manufacturing automation perspective, intelligent automation has been identified as a possible response to arising demands. The presented research aims to support the industrial uptake of intelligent automation into manufacturing businesses by quantifying risks at the early design stage and business case development. An early-stage decision-support framework for the implementation of intelligent automation in manufacturing businesses is presented in this thesis.The framework is informed by an extensive literature review, updated and verified with surveys and workshops to add to the knowledge base due to the rapid development of the associated technologies. A paradigm shift from cost to a risk-modelling perspective is proposed to provide a more flexible and generic approach applicable throughout the current technology landscape. The proposed probabilistic decision-support framework consists of three parts:• A clustering algorithm to identify the manufacturing functions in manual processes from task analysis to mitigate early-stage design uncertainties• A Bayesian Belief Network (BBN) informed by an expert elicitation via the DELPHI method, where the identified functions become the unit of analysis.• A Markov-Chain Monte-Carlo method modelling the effects of uncertainties on the critical success factors to address issues of factor interdependencies after expert elicitation.Based on the overall decision framework a toolbox was developed in Microsoft Excel. Five different case studies are used to test and validate the framework. Evaluation of the results derived from the toolbox from the industrial feedback suggests a positive validation for commercial use. The main contributions to knowledge in the presented thesis arise from the following four points:• Early-stage decision-support framework for business case evaluation of intelligent automation.• Translating manual tasks to automation function via a novel clustering approach• Application of a Markov-Chain Monte-Carlo Method to simulate correlation between decision criteria• Causal relationship among Critical Success Factors has been established from business and technical perspectives.The implications on practise might be promising. The feedback arising from the created tool was promising from the industry, and a practical realisation of the decision-support tool seems to be desired from an industrial point of view.With respect to further work, the decision-support tool might have established a ground to analyse a human task automatically for automation purposes. The established clustering mechanisms and the related attributes could be connected to sensorial data and analyse a manufacturing task autonomously without the subjective input of task analysis experts. To enable such an autonomous process, however, the psychophysiological understanding must be increased in the future.</div

Building expert systems: cognitive emulation.

Chapter 1 briefly introduces the concept of cognitive emulation, and outlines its current status. Chapter 2 reviews psychological research on human expert thinking. First, the study of expert thinking is placed in the context of modern cognitive psychology. Next, the principal methods and techniques employed by psychologists examining expert cognition are examined. The remainder of the chapter is given over to a review of the published literature on the nature and development of human expertise. Chapter 3 reviews the main arguments for and against cognitive emulation in expert system design. The tentative conclusion reached is that a significant degree of emulation is inevitable, but that a pure, unselective strategy of emulation is neither realistic nor desirable. Chapter 4 examines the prospects for cognitive emulation from a more pragmatic angle. Several factors are identified that represent constraints on the usefulness of a cognitive approach. However, a second set of factors is identified which should facilitate an emulation strategy - especially in the longer term. Some guidance is given on when to seriously consider adopting an emulation strategy. Chapter 5 presents a critical survey of expert system research that has already addressed the emulation issue. Six basic approaches to cognitive emulation are distinguished and evaluated. This helps draw out in more detail the implications of an emulation strategy for knowledge acquisition, knowledge representation and system architecture. The chapter concludes by discussing the issues that arise when different approaches to emulation are combined. Some guidance is offered on how this might be achieved. Chapter 6 summarizes the main themes and issues to have emerged, the design advice contained in the thesis, and the original contributions made by the thesis

Mercury Exposure Assessment of South River Floodplain Birds

The studies involved in this thesis expanded the current project being conducted in Dr. Newman’s laboratory that aimed to define and quantify the impacts of mercury movement in contaminated aquatic and terrestrial food webs in the South River watershed (Virginia, USA). This expansion involved a two phase study, which fulfilled the requirement of a master thesis. Previous research in our lab documented mercury biomagnification in the river itself and two floodplain locations on the South River watershed. Predictive models were built for mercury concentration in members of these food webs. These studies reached a preliminary conclusion that mercury biomagnification in members of floodplain food webs was faster than that of the aquatic food web. To substantiate this finding and further understand the factors that might produce the differences observed among floodplain locations, two additional floodplain locations were sampled and modeled in 2010. Overall, the models constructed in this study for predicting methylmercury were superior to models for total mercury or the percentage of the mercury present as methylmercury. Including previous models for other sites, four of five attempted methylmercury models based on δ15N met the criterion for useful prediction. For the floodplain models, thermoregulatory strategy was found to have substantial influence on mercury concentrations of food web members. The food web biomagnification factors for the four floodplain locations were consistently higher than that of the contiguous aquatic food web. The second phase of this research focused on description and determination of current mercury exposure to adults of three avian species during nesting on the South River floodplain and judgment of the risk of harmful mercury exposure to these species by comparing the mercury exposure distributions to published toxicity test results. This study incorporated a formal expert elicitation involving a modified Delphi framework and a Monte Carlo simulation to accomplish a probabilistic risk assessment. Simulations from this study predicted the probability that an adult bird during breeding season would ingest harmful amounts of mercury during daily foraging and also the probability that the average mercury ingestion rate for the breeding season of an adult bird would exceed published rates found to cause harm to other birds (\u3e100 ng total Hg/g body weight per day).The probabilities that these species’ averaged ingestion rates exceeded the threshold value were all less than 0.01

The management of risks in international infrastructural projects

In spite of the nature of construction contract risks, the variability involved in their outcomes and the potential benefits that applying rigorous and probabilistic approaches offers the analysis of such risks, existing predominant practices continue to involve the use of risk assessment and analysis approaches that are often arbitrary, illogical, inadequate, misleading and subject to considerable personal perceptions and biases of the "solo" analyst. The lack of rigour and systematic approach is often blamed on the possible high cost of pursuing a rigorous process and the unavailability of relative frequency data on the separate risks. The practice of using lump sum or percentage contingency, individual approaches to risk analysis and at best three-point or triangular distributions for risk analysis have thus persisted even though evidence from other industries suggests that rigorous and probabilistic approaches could be applied to construction contract risks.This thesis aims to conduct a review and survey to establish the appropriateness of the types of risk management techniques currently used in the construction industry, to investigate risk perception in construction and its impact on project performance, and to develop a procedural model for the elicitation of expert opinions about risks that minimises the adverse effects of risk perception on individual estimates of risk, and provides these opinions as input variables to the rigorous and probabilistic analysis of contractual risks. The work is a cross-cultural study, applying mail questionnaire surveys, interviews, Delphi and Vignette techniques, and analyzing risk management approaches and applications of the elicitation model developed by the study in both United Kingdom and Ghana. The data generated by the elicitation model are analysed using relative likelihood methods to develop subjective prior probability distributions for use as input variables in the Bayesian analysis of contractual risks in construction.The study concludes that although relative frequency data are often unavailable for contractual risks, existing predominant practices for contractual risk analysis are inappropriate for the nature of contractual risks. Furthermore, individual perceptions about risks significantly affect expert judgements about risks (and consequently project performance) in spite of their expertise. Using the expert elicitation model developed by the study and the analytical approaches applied, it is possible to capture, encode and aggregate the knowledge and experiences of a group of relevant experts to derive probability distribution functions of contractual risks to be applied as input variables to a Bayesian analysis of contractual risks, and thereby achieve a more appropriate, systematic and rigorous approach to contractual risk analysis. Evidence from the study also indicates that this approach need not involve any significantly high costs as the analysis can be done using standard spreadsheet software and add-in programmes that companies already have on their computer systems.Recommendations are thus made for the use of expert team approaches and the elicitation model developed in the study in the management of contractual risks. In addition, implications on existing types of contract, risk management education and further research are highlighted

Uncertainty analysis in product service system: Bayesian network modelling for availability contract

There is an emerging trend of manufacturing companies offering combined products and services to customers as integrated solutions. Availability contracts are an apt instance of such offerings, where product use is guaranteed to customer and is enforced by incentive-penalty schemes. Uncertainties in such an industry setting, where all stakeholders are striving to achieve their respective performance goals and at the same time collaborating intensively, is increased. Understanding through-life uncertainties and their impact on cost is critical to ensure sustainability and profitability of the industries offering such solutions. In an effort to address this challenge, the aim of this research study is to provide an approach for the analysis of uncertainties in Product Service System (PSS) delivered in business-to-business application by specifying a procedure to identify, characterise and model uncertainties with an emphasis to provide decision support and prioritisation of key uncertainties affecting the performance outcomes. The thesis presents a literature review in research areas which are at the interface of topics such as uncertainty, PSS and availability contracts. From this seven requirements that are vital to enhance the understanding and quantification of uncertainties in Product Service System are drawn. These requirements are synthesised into a conceptual uncertainty framework. The framework prescribes four elements, which include identifying a set of uncertainties, discerning the relationships between uncertainties, tools and techniques to treat uncertainties and finally, results that could ease uncertainty management and analysis efforts. The conceptual uncertainty framework was applied to an industry case study in availability contracts, where each of the four elements was realised. This application phase of the research included the identification of uncertainties in PSS, development of a multi-layer uncertainty classification, deriving the structure of Bayesian Network and finally, evaluation and validation of the Bayesian Network. The findings suggest that understanding uncertainties from a system perspective is essential to capture the network aspect of PSS. This network comprises of several stakeholders, where there is increased flux of information and material flows and this could be effectively represented using Bayesian Networks