Eliciting Information From Multiple Experts

A decision maker has to elicit information from informed experts regarding the desirability of a certain action from experts who share similar preferences which differ significantly from those of the decision maker. The question is how much information the decision maker can elicit, despite the difference in interests. The focus here is on ways in which the decision maker can take advantage of the multiplicity of experts. if the decision maker cannot commit to a mechanism and there is no communication among the experts, then no useful information is elicited from the experts in the equilibrium. If the experts can be partitioned into groups such that the members of each group can communicate with each other before they report their information to the decision maker, then more information can be elicited. Obviously, if all experts are allowed to communicate, they can be induced to reveal the relevant information, at least, when their aggregate information makes it desirable for them to undertake the project. The more interesting observation is that, if communication among the experts can be restricted to certain subsets, then even more information can be elicited. Finally, if the decision maker can commit to a mechanism, the information elicited in some cases is sufficient to implement the decision maker's best outcome in all but one state. All these observation make straightforward use of the idea that experts choose their report with the understanding that it matters only when they are pivotal.

Eliciting Uncertain Resilience Information for Risk Mitigation

The literature of risk, mitigation, and resilience is rich in classifications and recommendations. The missing link is evaluation: ideally, data based; initially, based on expert judgment. We present a novel approach for eliciting probability distributions describing mitigation effectiveness. This approach can be used by subject matter experts (SMEs) who are not specialists in mathematics or engineering. A visual interface permits each expert to sketch a distribution by moving five colored dots on the user interface. The engine can weight and combine estimates from several SMEs into an aggregate density function suitable for presentation, and an aggregate cumulated distribution for use in Monte Carlo simulations. Additional supporting software adapts the tool for real-time support of virtual Delphi-type sessions involving multiple distributed experts. Use of the tool in a study aimed at controlling information and communication technology supply chain risks yields valuable information on those threats, and on the tool itself

Using XML and XSLT for flexible elicitation of mental-health risk knowledge

Current tools for assessing risks associated with mental-health problems require assessors to make high-level judgements based on clinical experience. This paper describes how new technologies can enhance qualitative research methods to identify lower-level cues underlying these judgements, which can be collected by people without a specialist mental-health background. Methods and evolving results: Content analysis of interviews with 46 multidisciplinary mental-health experts exposed the cues and their interrelationships, which were represented by a mind map using software that stores maps as XML. All 46 mind maps were integrated into a single XML knowledge structure and analysed by a Lisp program to generate quantitative information about the numbers of experts associated with each part of it. The knowledge was refined by the experts, using software developed in Flash to record their collective views within the XML itself. These views specified how the XML should be transformed by XSLT, a technology for rendering XML, which resulted in a validated hierarchical knowledge structure associating patient cues with risks. Conclusions: Changing knowledge elicitation requirements were accommodated by flexible transformations of XML data using XSLT, which also facilitated generation of multiple data-gathering tools suiting different assessment circumstances and levels of mental-health knowledge

Evaluation of elicitation methods to quantify Bayes linear models

The Bayes linear methodology allows decision makers to express their subjective beliefs and adjust these beliefs as observations are made. It is similar in spirit to probabilistic Bayesian approaches, but differs as it uses expectation as its primitive. While substantial work has been carried out in Bayes linear analysis, both in terms of theory development and application, there is little published material on the elicitation of structured expert judgement to quantify models. This paper investigates different methods that could be used by analysts when creating an elicitation process. The theoretical underpinnings of the elicitation methods developed are explored and an evaluation of their use is presented. This work was motivated by, and is a precursor to, an industrial application of Bayes linear modelling of the reliability of defence systems. An illustrative example demonstrates how the methods can be used in practice

Eliciting Domain Knowledge Using Conceptual Metaphors: A Case Study from Music Interaction

Interaction design for domains that involve complex abstractions can prove challenging. This problem is particularly acute in domains where the intricate nature of domain-specific knowledge can be difficult for even the most experienced expert to conceptualise or articulate. One promising solution to the problem of representing complex domain abstractions involves the use of conceptual metaphors. Previous applications of conceptual metaphors to abstract domains have yielded encouraging results. However, the design of appropriate methods for eliciting conceptual metaphors for the purposes of informing interaction design remains an open question. In this paper, we report on a series of studies carried out to elicit conceptual metaphors from domain experts, using music as a case study, reflecting on the benefits and drawbacks of each approach

Elicitation of structured engineering judgement to inform a focussed FMEA

The practical use of Failure Mode and Effects Analysis (FMEA) has been criticised because it is often implemented too late and in a manner that does not allow information to be fed-back to inform the product design. Lessons learnt from the use of elicitation methods to gather structured expert judgement about engineering concerns for a new product design has led to an enhancement of the approach for implementing design and process FMEA. We refer to this variant as a focussed FMEA since the goal is to enable relevant engineers to contribute to the analysis and to act upon the outcomes in such a way that all activities focus upon the design needs. The paper begins with a review of the proposed process to identify and quantify engineering concerns. The pros and cons of using elicitation methods, originally designed to support construction of a Bayesian prior, to inform a focussed FMEA are analysed and a comparison of the proposed process in relation to the existing standards is made. An industrial example is presented to illustrate customisation of the process and discuss the impact on the design process

Expert Elicitation for Reliable System Design

This paper reviews the role of expert judgement to support reliability assessments within the systems engineering design process. Generic design processes are described to give the context and a discussion is given about the nature of the reliability assessments required in the different systems engineering phases. It is argued that, as far as meeting reliability requirements is concerned, the whole design process is more akin to a statistical control process than to a straightforward statistical problem of assessing an unknown distribution. This leads to features of the expert judgement problem in the design context which are substantially different from those seen, for example, in risk assessment. In particular, the role of experts in problem structuring and in developing failure mitigation options is much more prominent, and there is a need to take into account the reliability potential for future mitigation measures downstream in the system life cycle. An overview is given of the stakeholders typically involved in large scale systems engineering design projects, and this is used to argue the need for methods that expose potential judgemental biases in order to generate analyses that can be said to provide rational consensus about uncertainties. Finally, a number of key points are developed with the aim of moving toward a framework that provides a holistic method for tracking reliability assessment through the design process.Comment: This paper commented in: [arXiv:0708.0285], [arXiv:0708.0287], [arXiv:0708.0288]. Rejoinder in [arXiv:0708.0293]. Published at http://dx.doi.org/10.1214/088342306000000510 in the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org