Eliciting density ratio classes

AbstractThe probability distributions of uncertain quantities needed for predictive modelling and decision support are frequently elicited from subject matter experts. However, experts are often uncertain about quantifying their beliefs using precise probability distributions. Therefore, it seems natural to describe their uncertain beliefs using sets of probability distributions. There are various possible structures, or classes, for defining set membership of continuous random variables. The Density Ratio Class has desirable properties, but there is no established procedure for eliciting this class. Thus, we propose a method for constructing Density Ratio Classes that builds on conventional quantile or probability elicitation, but allows the expert to state intervals for these quantities. Parametric shape functions, ideally also suggested by the expert, are then used to bound the nonparametric set of shapes of densities that belong to the class and are compatible with the stated intervals. This leads to a natural metric for the size of the class based on the ratio of the total areas under upper and lower bounding shape functions. This ratio will be determined by the characteristics of the shape functions, the scatter of the elicited values, and the explicit expert imprecision, as characterized by the width of the stated intervals. We provide some examples, both didactic and real, and conclude with recommendations for the further development and application of the Density Ratio Class

Applying multiple criteria decision analysis to comparative benefit-risk assessment: choosing among statins in primary prevention

Decision makers in different health care settings need to weigh the benefits and harms of alternative treatment strategies. Such health care decisions include marketing authorization by regulatory agencies, practice guideline formulation by clinical groups, and treatment selection by prescribers and patients in clinical practice. Multiple criteria decision analysis (MCDA) is a family of formal methods that help make explicit the tradeoffs that decision makers accept between the benefit and risk outcomes of different treatment options. Despite the recent interest in MCDA, certain methodological aspects are poorly understood. This paper presents 7 guidelines for applying MCDA in benefitrisk assessment and illustrates their use in the selection of a statin drug for the primary prevention of cardiovascular disease. We provide guidance on the key methodological issues of how to define the decision problem, how to select a set of nonoverlapping evaluation criteria, how to synthesize and summarize the evidence, how to translate relative measures to absolute ones that permit comparisons between the criteria, how to define suitable scale ranges, how to elicit partial preference information from the decision makers, and how to incorporate uncertainty in the analysis. Our example on statins indicates that fluvastatin is likely to be the most preferred drug by our decision maker and that this result is insensitive to the amount of preference information incorporated in the analysis

Reasons and Means to Model Preferences as Incomplete

Literature involving preferences of artificial agents or human beings often assume their preferences can be represented using a complete transitive binary relation. Much has been written however on different models of preferences. We review some of the reasons that have been put forward to justify more complex modeling, and review some of the techniques that have been proposed to obtain models of such preferences

Does Consistency Predict Accuracy of Beliefs?: Economists Surveyed About PSA

Subjective beliefs and behavior regarding the Prostate Specific Antigen (PSA) test for prostate cancer were surveyed among attendees of the 2006 meeting of the American Economic Association. Logical inconsistency was measured in percentage deviations from a restriction imposed by Bayes’ Rule on pairs of conditional beliefs. Economists with inconsistent beliefs tended to be more accurate than average, and consistent Bayesians were substantially less accurate. Within a loss function framework, we look for and cannot find evidence that inconsistent beliefs cause economic losses. Subjective beliefs about cancer risks do not predict PSA testing decisions, but social influences do.logical consistency, predictive accuracy, elicitation, non-Bayesian, ecological rationality

Does consistency predict accuracy of beliefs?: Economists surveyed about PSA

Subjective beliefs and behavior regarding the Prostate Specific Antigen (PSA) test for prostate cancer were surveyed among attendees of the 2006 meeting of the American Economic Association. Logical inconsistency was measured in percentage deviations from a restriction imposed by Bayes’ Rule on pairs of conditional beliefs. Economists with inconsistent beliefs tended to be more accurate than average, and consistent Bayesians were substantially less accurate. Within a loss function framework, we look for and cannot find evidence that inconsistent beliefs cause economic losses. Subjective beliefs about cancer risks do not predict PSA testing decisions, but social influences do.logical consistency, predictive accuracy, elicitation, non-Bayesian, ecological rationality

Robust detection of exotic infectious diseases in animal herds: A comparative study of three decision methodologies under severe uncertainty

When animals are transported and pass through customs, some of them may have dangerous infectious diseases. Typically, due to the cost of testing, not all animals are tested: a reasonable selection must be made. How to test effectively whilst avoiding costly disease outbreaks? First, we extend a model proposed in the literature for the detection of invasive species to suit our purpose. Secondly, we explore and compare three decision methodologies on the problem at hand, namely, Bayesian statistics, info-gap theory and imprecise probability theory, all of which are designed to handle severe uncertainty. We show that, under rather general conditions, every info-gap solution is maximal with respect to a suitably chosen imprecise probability model, and that therefore, perhaps surprisingly, the set of maximal options can be inferred at least partly---and sometimes entirely---from an info-gap analysis

Robust decision analysis under severe uncertainty and ambiguous tradeoffs: an invasive species case study

Bayesian decision analysis is a useful method for risk management decisions, but is limited in its ability to consider severe uncertainty in knowledge, and value ambiguity in management objectives. We study the use of robust Bayesian decision analysis to handle problems where one or both of these issues arise. The robust Bayesian approach models severe uncertainty through bounds on probability distributions, and value ambiguity through bounds on utility functions. To incorporate data, standard Bayesian updating is applied on the entire set of distributions. To elicit our expert's utility representing the value of different management objectives, we use a modified version of the swing weighting procedure that can cope with severe value ambiguity. We demonstrate these methods on an environmental management problem to eradicate an alien invasive marmorkrebs recently discovered in Sweden, which needed a rapid response despite substantial knowledge gaps if the species was still present (i.e., severe uncertainty) and the need for difficult tradeoffs and competing interests (i.e., value ambiguity). We identify that the decision alternatives to drain the system and remove individuals in combination with dredging and sieving with or without a degradable biocide, or increasing pH, are consistently bad under the entire range of probability and utility bounds. This case study shows how robust Bayesian decision analysis provides a transparent methodology for integrating information in risk management problems where little data are available and/or where the tradeoffs are ambiguous

Relating Imprecise Representations of imprecise Probabilities

International audienceThere exist many practical representations of probability families that make them easier to handle. Among them are random sets, possibility distributions, probability intervals, Ferson's p-boxes and Neumaier's clouds. Both for theoretical and practical considerations, it is important to know whether one representation has the same expressive power than other ones, or can be approximated by other ones. In this paper, we mainly study the relationships between the two latter representations and the three other ones

Robust analysis of uncertainty in scientific assessments

Uncertainty refers to any limitation in knowledge. Identifying and characterizing uncertainty in conclusions is important to ensure transparency and avoid over or under confidence in scientific assessments. Quantitative expressions of uncertainty are less ambiguous compared to uncertainty expressed qualitatively, or not at all. Subjective probability is an example of a quantitative expression of epistemic uncertainty, which combined with Bayesian inference makes it possible to integrate evidence and characterizes uncertainty in quantities of interest. This thesis contributes to the understanding and implementation of robust Bayesian analysis as a way to integrate expert judgment and data into assessments and quantify uncertainty by bounded probability. The robust Bayesian framework is based on sets of probability for epistemic uncertainty, where precise probability is seen as a special case. This thesis covers applications relevant for scientific assessments, including evidence synthesis and quantitative risk assessment.Paper I proposes to combine two sampling methods: iterative importance sampling and Markov chain Monte Carlo (MCMC) sampling, for quantifying uncertainty by bounded probability when Bayesian updating requires MCMC sampling. This opens up for robust Bayesian analysis to be applied to complex statistical models. To achieve this, an effective sample size of importance sampling that accounts for correlated MCMC samples is proposed. For illustration, the proposed method is applied to estimate the overall effect with bounded probability in a published meta-analysis within the Collaboration for Environmental Evidence on the effect of biomanipulation on freshwater lakes.Paper II demonstrates robust Bayesian analysis as a way to quantify uncertainty in a quantity of interest by bounded probability, and explicitly distinguishes between epistemic and aleatory uncertainty in the assessment and learn parameters by integrating evidence into the model. Robust Bayesian analysis is described as a generalization of Bayesian analysis, including Bayesian analysis through precise probability as a special case. Both analyses are applied to an intake assessment.Paper III describes a way to consider uncertainty arising from ignorance or ambiguity about bias terms in a quantitative bias analysis by characterizing bias with imprecision. This is done by specifying bias with a set of bias terms and use robust Bayesian analysis to estimate the overall effect in the meta-analysis. The approach provides a structured framework to transform qualitative judgments concerning risk of biases into quantitative expressions of uncertainty in quantitative bias analysis.Paper IV compares the effect of different diversified farming practices on biodiversity and crop yields. This is done by applying a Bayesian network meta-analysis to a new public global database from a systematic protocol on diversified farming. A portfolio analysis calibrated by the network meta-analyses showed that uncertainty about the mean performance is large compared to the variability in performance across different farms