35 research outputs found

    Automated Update Tools To Augment the Wisdom of Crowds in Geopolitical Forecasting

    Get PDF
    Despite the importance of predictive judgments, individual human forecasts are frequently less accurate than those of even simple prediction algorithms. At the same time, not all forecasts are amenable to algorithmic prediction. Here, we describe the evaluation of an automated prediction tool that enabled participants to create simple rules that monitored relevant indicators (e.g., commodity prices) to automatically update forecasts. We examined these rules in both a pool of previous participants in a geopolitical forecasting tournament (Study 1) and a naĂŻve sample recruited from Mechanical Turk (Study 2). Across the two studies, we found that automated updates tended to improve forecast accuracy relative to initial forecasts and were comparable to manual updates. Additionally, making rules improved the accuracy of manual updates. Crowd forecasts likewise benefitted from rule-based updates. However, when presented with the choice of whether to accept, reject or adjust an automatic forecast update, participants showed little ability to discriminate between automated updates that were harmful versus beneficial to forecast accuracy. Simple prospective rule-based tools are thus able to improve forecast accuracy by offering accurate and efficient updates, but ensuring forecasters make use of tools remains a challenge

    Deep Neural Ranking for Crowdsourced Geopolitical Event Forecasting

    Get PDF
    There are many examples of “wisdom of the crowd” effects in which the large number of participants imparts confidence in the collective judgment of the crowd. But how do we form an aggregated judgment when the size of the crowd is limited? Whose judgments do we include, and whose do we accord the most weight? This paper considers this problem in the context of geopolitical event forecasting, where volunteer analysts are queried to give their expertise, confidence, and predictions about the outcome of an event. We develop a forecast aggregation model that integrates topical information about a question, meta-data about a pair of forecasters, and their predictions in a deep siamese neural network that decides which forecasters’ predictions are more likely to be close to the correct response. A ranking of the forecasters is induced from a tournament of pair-wise forecaster comparisons, with the ranking used to create an aggregate forecast. Preliminary results find the aggregate prediction of the best forecasters ranked by our deep siamese network model consistently beats typical aggregation techniques by Brier score

    The Case of the Cognitive (Opti)miser: Electrophysiological Correlates of Working Memory Maintenance Predict Demand Avoidance

    No full text
    People are often considered cognitive misers. When given a free choice between two tasks, people tend to choose tasks requiring less cognitive effort. Such demand avoidance (DA) is associated with cognitive control, but it is still not clear to what extent individual differences in cognitive control can account for variations in DA. We sought to elucidate the relation between cognitive control and cognitive effort preferences by investigating the extent to which sustained neural activity in a task requiring cognitive control is correlated with DA. We hypothesized that neural measures of efficient filtering will predict individual variations in demand preferences. To test this hypothesis, we had participants perform a delayed-match-tosample paradigm with their ERPs recorded, as well as a separate behavioral demand-selection task. We focused on the ERP correlates of cognitive filtering efficiency (CFE)--the ability to ignore task-irrelevant distractors during working memory maintenance--as it manifests in a modulation of the contralateral delay activity, an ERP correlate of cognitive control. As predicted, we found a significant positive correlation between CFE and DA. Individuals with high CFE tended to be significantly more demand avoidant than their low-CFE counterparts. Low-CFE individuals, in comparison, did not form distinct cognitive effort preferences. Overall, our results suggest that cognitive control over the contents of visual working memory contribute to individual differences in the expression of cognitive effort preferences. This further implies that these observed preferences are the product of sensitivity to cognitive task demands

    The Case of the Cognitive (Opti)miser: Electrophysiological Correlates of Working Memory Maintenance Predict Demand Avoidance

    No full text
    People are often considered cognitive misers. When given a free choice between two tasks, people tend to choose tasks requiring less cognitive effort. Such demand avoidance (DA) is associated with cognitive control, but it is still not clear to what extent individual differences in cognitive control can account for variations in DA. We sought to elucidate the relation between cognitive control and cognitive effort preferences by investigating the extent to which sustained neural activity in a task requiring cognitive control is correlated with DA. We hypothesized that neural measures of efficient filtering will predict individual variations in demand preferences. To test this hypothesis, we had participants perform a delayed-match-tosample paradigm with their ERPs recorded, as well as a separate behavioral demand-selection task. We focused on the ERP correlates of cognitive filtering efficiency (CFE)--the ability to ignore task-irrelevant distractors during working memory maintenance--as it manifests in a modulation of the contralateral delay activity, an ERP correlate of cognitive control. As predicted, we found a significant positive correlation between CFE and DA. Individuals with high CFE tended to be significantly more demand avoidant than their low-CFE counterparts. Low-CFE individuals, in comparison, did not form distinct cognitive effort preferences. Overall, our results suggest that cognitive control over the contents of visual working memory contribute to individual differences in the expression of cognitive effort preferences. This further implies that these observed preferences are the product of sensitivity to cognitive task demands

    Hybrid Forecasting: Triage Studies

    No full text

    Effects of Choice Restriction on Accuracy and User Experience in an Internet-Based Geopolitical Forecasting Task

    No full text
    Large-scale geopolitical forecasting tournaments have emerged in recent years as effective testbeds for conducting research into novel forecasting tools and methods. A challenge of such tournaments involves the distribution of forecasting load across forecasters, since there are often more forecasting questions than an individual forecaster can answer. Intelligent load distribution, or triage, may therefore be helpful in ensuring that all questions have sufficient numbers of forecasts to benefit from crowd-based aggregation and that individual forecasters are matched to the questions for which they are best suited. A possible downside of triage, however, is that it restricts the choices of forecasters, potentially degrading motivation and accuracy. In two studies involving pools of novice forecasters recruited online, we examined the impact of limiting forecaster choice on forecasters’ accuracy and subjective experience, including motivation. In Study 1, we tested the impact of restricted choice by comparing the forecasting accuracy and subjective experience of users who perceived they did or did not have choice in the questions they forecasted. In Study 2, we further tested the impact of restricted choice by providing users with different menu sizes of questions from which to choose. In both studies, we found no evidence that limiting forecaster choice adversely affected forecasting accuracy or subjective experience. This suggests that in large-scale forecasting tournaments, it may be possible to implement choice-limiting triage strategies without sacrificing individual accuracy and motivation

    Effects of Choice Restriction on Accuracy and User Experience in an Internet-Based Geopolitical Forecasting Task

    No full text
    Large-scale geopolitical forecasting tournaments have emerged in recent years as effective testbeds for conducting research into novel forecasting tools and methods. A challenge of such tournaments involves the distribution of forecasting load across forecasters, since there are often more forecasting questions than an individual forecaster can answer. Intelligent load distribution, or triage, may therefore be helpful in ensuring that all questions have sufficient numbers of forecasts to benefit from crowd-based aggregation and that individual forecasters are matched to the questions for which they are best suited. A possible downside of triage, however, is that it restricts the choices of forecasters, potentially degrading motivation and accuracy. In two studies involving pools of novice forecasters recruited online, we examined the impact of limiting forecaster choice on forecasters’ accuracy and subjective experience, including motivation. In Study 1, we tested the impact of restricted choice by comparing the forecasting accuracy and subjective experience of users who perceived they did or did not have choice in the questions they forecasted. In Study 2, we further tested the impact of restricted choice by providing users with different menu sizes of questions from which to choose. In both studies, we found no evidence that limiting forecaster choice adversely affected forecasting accuracy or subjective experience. This suggests that in large-scale forecasting tournaments, it may be possible to implement choice-limiting triage strategies without sacrificing individual accuracy and motivation
    corecore