Can groups improve expert economic and financial forecasts?

Economic and financial forecasts are important for business planning and government policy but are notoriously challenging. We take advantage of recent advances in individual and group judgement, and a data set of economic and financial forecasts compiled over 25 years, consisting of multiple individual and institutional estimates, to test the claim that nominal groups will make more accurate economic and financial forecast than individuals. We validate the forecasts using the subsequent published (real) outcomes, explore the performance of nominal groups against institutions, identify potential superforecasters and discuss the benefits of implementing structured judgment techniques to improve economic and financial forecasts

Improving expert forecasts in reliability. Application and evidence for structured elicitation protocols

Quantitative expert judgementsare used in reliability assessmentsto informcritically important decisions. Structured elicitation protocols have been advocated to improveexpert judgements, yet their application in reliability ischallenged by a lack of examples or evidence that they improve judgements. This paper aims to overcome these barriers. We present a case study where two world-leading protocols, the IDEA protocol and the Classical Model were combined and applied by the Australian Department of Defence for a reliability assessment. We assess the practicality of the methods, and the extent to which they improve judgements. The average expert was extremely overconfident, with 90% credible intervals containing the true realisation 36% of the time. However,steps contained inthe protocols substantially improvedjudgements. In particular, an equal weighted aggregation of individual judgements, and the inclusion ofa discussion phase and revised estimate helped to improve calibration, statistical accuracy and the Classical Model score. Further improvements in precision and information were made via performance weighted aggregation. This paper provides useful insights into the application of structured elicitation protocols for reliability andthe extent to which judgements are improved. The findings raise concerns about existing practices for utilising experts in reliability assessments and suggest greater adoption of structured protocols is warranted. We encourage the reliability community to further develop examples and insights

The value of performance weights and discussion in aggregated expert judgements

In risky situations characterized by imminent decisions, scarce resources, and insufficient data, policymakers rely on experts to estimate model parameters and their associated uncertainties. Different elicitation and aggregation methods can vary substantially in their efficacy and robustness. While it is generally agreed that biases in expert judgments can be mitigated using structured elicitations involving groups rather than individuals, there is still some disagreement about how to best elicit and aggregate judgments. This mostly concerns the merits of using performance‐based weighting schemes to combine judgments of different individuals (rather than assigning equal weights to individual experts), and the way that interaction between experts should be handled. This article aims to contribute to, and complement, the ongoing discussion on these topics

Online training courses on Expert Knowledge Elicitation (EKE)

This report summarises the training courses delivered under the contract OC/EFSA/AMU/2021/02 EKE: “Develop and conduct online training courses on Expert Knowledge Elicitation (EKE)”. The objective of the courses was to develop and conduct online training courses on applying the methodology described in the EFSA Guidance on Expert Knowledge Elicitation in Food and Feed Safety Risk Assessment” for EFSA staff and experts, as well as corresponding experts from EU member states. In addition to the three standard EKE methods (Sheffield, Delphi and Cooke), the training included a semi-formal method of EKE. All these methods may be used when EKE is performed within an existing EFSA working group to support uncertainty analysis as outlined in “The principles and methods behind EFSA\u27s Guidance on Uncertainty Analysis in Scientific Assessment”. In total, 12 courses were organised: two on “Steering an Expert Knowledge Elicitation”, two on “Conduct of the Sheffield protocol for an EKE”, one on “Conduct of the Cooke protocol for an EKE”, one on “Conduct of the Delphi protocol for an EKE”, two on “Conduct of a Semi-formal EKE”, two on “Reporting an Expert Knowledge Elicitation” and two on “Writing an Evidence Dossier for an Expert Knowledge Elicitation”. The courses had in total 149 participants and received very good feedback from the participants with a mean value of 4.2 of 5 possible, considering all numerical questions in the feedback questionnaire. Recommendations for future activities on training EKE methodologies are provided

Predicting species and community responses to global change using structured expert judgement : an Australian mountain ecosystems case study

Conservation managers are under increasing pressure to make decisions about the allocation of finite resources to protect biodiversity under a changing climate. However, the impacts of climate and global change drivers on species are outpacing our capacity to collect the empirical data necessary to inform these decisions. This is particularly the case in the Australian Alps which has already undergone recent changes in climate and experienced more frequent large-scale bushfires. In lieu of empirical data, we used a structured expert elicitation method (the IDEA protocol) to estimate the abundance and distribution of nine vegetation groups and 89 Australian alpine and subalpine species by the year 2050. Experts predicted that most alpine vegetation communities would decline in extent by 2050; only woodlands and heathlands are predicted to increase in extent. Predicted species-level responses for alpine plants and animals were highly variable and uncertain. In general, alpine plants spanned the range of possible responses, with some expected to increase, decrease or not change in cover. By contrast, almost all animal species are predicted to decline or not change in abundance or elevation range; more species with water-centric life-cycles are expected to decline in abundance than other species. While long-term ecological data will always be the gold-standard in informing the future of biodiversity, the method and outcomes outlined here provide a pragmatic and coherent basis upon which to start informing conservation policy and management in the face of rapid change and paucity of data

A Three-Part Bayesian Network for Modeling Dwelling Fires and Their Impact upon People and Property.

In the United Kingdom, dwelling fires are responsible for the majority of all fire-related fatalities. The development of these incidents involves the interaction of a multitude of variables that combine in many different ways. Consequently, assessment of dwelling fire risk can be complex, which often results in ambiguity during fire safety planning and decision making. In this article, a three-part Bayesian network model is proposed to study dwelling fires from ignition through to extinguishment in order to improve confidence in dwelling fire safety assessment. The model incorporates both hard and soft data, delivering posterior probabilities for selected outcomes. Case studies demonstrate how the model functions and provide evidence of its use for planning and accident investigation

Mathematically aggregating experts' predictions of possible futures

Structured protocols offer a transparent and systematic way to elicit and combine/aggregate, probabilistic predictions from multiple experts. These judgements can be aggregated behaviourally or mathematically to derive a final group prediction. Mathematical rules (e.g., weighted linear combinations of judgments) provide an objective approach to aggregation. The quality of this aggregation can be defined in terms of accuracy, calibration and informativeness. These measures can be used to compare different aggregation approaches and help decide on which aggregation produces the “best” final prediction. When experts’ performance can be scored on similar questions ahead of time, these scores can be translated into performance-based weights, and a performance-based weighted aggregation can then be used. When this is not possible though, several other aggregation methods, informed by measurable proxies for good performance, can be formulated and compared. Here, we develop a suite of aggregation methods, informed by previous experience and the available literature. We differentially weight our experts’ estimates by measures of reasoning, engagement, openness to changing their mind, informativeness, prior knowledge, and extremity, asymmetry or granularity of estimates. Next, we investigate the relative performance of these aggregation methods using three datasets. The main goal of this research is to explore how measures of knowledge and behaviour of individuals can be leveraged to produce a better performing combined group judgment. Although the accuracy, calibration, and informativeness of the majority of methods are very similar, a couple of the aggregation methods consistently distinguish themselves as among the best or worst. Moreover, the majority of methods outperform the usual benchmarks provided by the simple average or the median of estimates

Assessment of the response of pollinator abundance to environmental pressures using structured expert elicitation

Policy-makers often need to rely on experts with disparate fields of expertise when making policy choices in complex, multi-faceted, dynamic environments such as those dealing with ecosystem services. For policy-makers wishing to make evidence-based decisions which will best support pollinator abundance and pollination services, one of the problems faced is how to access the information and evidence they need, and how to combine it to formulate and evaluate candidate policies. This is even more complex when multiple factors provide influence in combination. The pressures affecting the survival and pollination capabilities of honey bees (Apis mellifera), wild bees, and other pollinators are well documented, but incomplete. In order to estimate the potential effectiveness of various candidate policy choices, there is an urgent need to quantify the effect of various combinations of factors on the pollination ecosystem service. Using high-quality experimental evidence is the most robust approach, but key aspects of the system may not be amenable to experimentation or may be prohibitive based on cost, time and effort. In such cases, it is possible to obtain the required evidence by using structured expert elicitation, a method for quantitatively characterizing the state of knowledge about an uncertain quantity. Here we report and discuss the outputs of the novel use of a structured expert elicitation, designed to quantify the probability of good pollinator abundance given a variety of weather, disease, and habitat scenarios