Mathematically aggregating experts' predictions of possible futures

Structured protocols offer a transparent and systematic way to elicit and combine/aggregate, probabilistic predictions from multiple experts. These judgements can be aggregated behaviourally or mathematically to derive a final group prediction. Mathematical rules (e.g., weighted linear combinations of judgments) provide an objective approach to aggregation. The quality of this aggregation can be defined in terms of accuracy, calibration and informativeness. These measures can be used to compare different aggregation approaches and help decide on which aggregation produces the “best” final prediction. When experts’ performance can be scored on similar questions ahead of time, these scores can be translated into performance-based weights, and a performance-based weighted aggregation can then be used. When this is not possible though, several other aggregation methods, informed by measurable proxies for good performance, can be formulated and compared. Here, we develop a suite of aggregation methods, informed by previous experience and the available literature. We differentially weight our experts’ estimates by measures of reasoning, engagement, openness to changing their mind, informativeness, prior knowledge, and extremity, asymmetry or granularity of estimates. Next, we investigate the relative performance of these aggregation methods using three datasets. The main goal of this research is to explore how measures of knowledge and behaviour of individuals can be leveraged to produce a better performing combined group judgment. Although the accuracy, calibration, and informativeness of the majority of methods are very similar, a couple of the aggregation methods consistently distinguish themselves as among the best or worst. Moreover, the majority of methods outperform the usual benchmarks provided by the simple average or the median of estimates

Development and validation of a diabetes-specific health state classification system and valuation function based on the multi-attribute theory

Preference-Based Measures of Health (PBMH) provide \u27preference\u27 or \u27utility\u27 weights that enable the calculation of quality-adjusted life years for the economic evaluations of interventions. The Diabetes Utility Index (DUI) was developed as a two-page, self-administered diabetes-specific PBMH that can replace expensive time-consuming interviews with patients to estimate their health state utilities. Inputs from theory, an existing diabetes-specific measure of quality of life, and statistical analyses were submitted to a clinical expert panel. After three rounds of pilot surveys (n1=52, n2=65, n3=111) at primary care clinics in Morgantown, WV, five attributes and severity categories for each attribute were finalized on the basis of the results of Rasch Analysis and consultations with the panel. The final attributes were: \u27Physical Ability & Energy\u27, \u27Relationships\u27, \u27Mood & Feelings\u27, \u27Enjoyment of Diet\u27, and \u27Satisfaction with Management of diabetes\u27. The next step involved obtaining preferences for health states based on combinations of DUI attributes and severity levels from 100 individuals with diabetes, recruited from primary care and community settings in and around Morgantown, WV, in hour-long one-on-one interviews. These health states were anchor states, single-attribute level states including corner states, and marker states. The interviews provided data to calculate a Multi-Attribute Utility Function (MAUF) that calculates utilities for any of the 768 health states that can be defined by the DUI, on a scale where 1.00=Perfect Health and 0.00=the all worse \u27Pits\u27 state, from respondents\u27 answers to its five questions. In addition to an overall index score, attribute-level preference scores were also calculable by the function. Finally, a validation survey was conducted in collaboration with the West Virginia University (WVU) Diabetes Institute. For concurrent and construct validation purposes, the DUI was mailed to individuals with diabetes along with generic PBMH like the EuroQol EQ-5D, the SF-6D and other patient-reported outcomes measures like the Diabetes Symptoms Checklist-Revised, the Short Form 12 (SF-12) and the Well-Being Questionnaire (W-BQ12), and their surveys responses (n=396) were merged with a clinical database consisting of ICD-9 diagnosis codes. The DUI utilities were found to be largely free of socio-demographic effects and its scores were well distributed between 0.00 and 1.00. The DUI moderately correlated with generic PBMH and distinguished between severity groups based on diabetes symptoms and complications. The scoring function of the DUI calculated utilities favorably compared against cardinal Standard Gamble utilities obtained directly from patients for three DUI health states. These results show evidence of the feasibility and validity of the DUI. Further research is suggested to demonstrate the generalizability of these findings, to study the responsiveness of the DUI, and to examine the clinical meaningfulness of the DUI change scores

Suburban watershed nitrogen retention : estimating the effectiveness of stormwater management structures

Excess nitrogen (N) is a primary driver of freshwater and coastal eutrophication globally, and urban stormwater is a rapidly growing source of N pollution. Stormwater best management practices (BMPs) are used widely to remove excess N from runoff in urban and suburban areas, and are expected to perform under a wide variety of environmental conditions. Yet the capacity of BMPs to retain excess N varies; and both the variation and the drivers thereof are largely unknown, hindering the ability of water resource managers to meet water quality targets in a cost-effective way. Here, we use structured expert judgment (SEJ), a performance-weighted method of expert elicitation, to quantify the uncertainty in BMP performance under a range of site-specific environmental conditions and to estimate the extent to which key environmental factors influence variation in BMP performance. We hypothesized that rain event frequency and magnitude, BMP type and size, and physiographic province would significantly influence the experts’ estimates of N retention by BMPs common to suburban Piedmont and Coastal Plain watersheds of the Chesapeake Bay region. Expert knowledge indicated wide uncertainty in BMP performance, with N removal efficiencies ranging from 40%. Experts believed that the amount of rain was the primary identifiable source of variability in BMP efficiency, which is relevant given climate projections of more frequent heavy rain events in the mid-Atlantic. To assess the extent to which those projected changes might alter N export from suburban BMPs and watersheds, we combined downscaled estimates of rainfall with distributions of N loads for different-sized rain events derived from our elicitation. The model predicted higher and more variable N loads under a projected future climate regime, suggesting that current BMP regulations for reducing nutrients may be inadequate in the future

Use of expert elicitation to assign weights to climate and hydrological models in climate impact studies

Various methods are available for assessing uncertainties in climate impact studies. Among such methods, model weighting by expert elicitation is a practical way to provide a weighted ensemble of models for specific real-world impacts. The aim is to decrease the influence of improbable models in the results and easing the decisionmaking process. In this study both climate and hydrological models are analysed, and the result of a research experiment is presented using model weighting with the participation of six climate model experts and six hydrological model experts. For the experiment, seven climate models are a priori selected from a larger EURO-CORDEX (Coordinated Regional Downscaling Experiment – European Domain) ensemble of climate models, and three different hydrological models are chosen for each of the three European river basins. The model weighting is based on qualitative evaluation by the experts for each of the selected models based on a training material that describes the overall model structure and literature about climate models and the performance of hydrological models for the present period. The expert elicitation process follows a three-stage approach, with two individual rounds of elicitation of probabilities and a final group consensus, where the experts are separated into two different community groups: a climate and a hydrological modeller group. The dialogue reveals that under the conditions of the study, most climate modellers prefer the equal weighting of ensemble members, whereas hydrological-impact modellers in general are more open for assigning weights to different models in a multi-model ensemble, based on model performance and model structure. Climate experts are more open to exclude models, if obviously flawed, than to put weights on selected models in a relatively small ensemble. The study shows that expert elicitation can be an efficient way to assign weights to different hydrological models and thereby reduce the uncertainty in climate impact. However, for the climate model ensemble, comprising seven models, the elicitation in the format of this study could only re-establish a uniform weight between climate models.European Commission European Commission Joint Research Centre 69046