Automated Update Tools To Augment the Wisdom of Crowds in Geopolitical Forecasting

Despite the importance of predictive judgments, individual human forecasts are frequently less accurate than those of even simple prediction algorithms. At the same time, not all forecasts are amenable to algorithmic prediction. Here, we describe the evaluation of an automated prediction tool that enabled participants to create simple rules that monitored relevant indicators (e.g., commodity prices) to automatically update forecasts. We examined these rules in both a pool of previous participants in a geopolitical forecasting tournament (Study 1) and a naïve sample recruited from Mechanical Turk (Study 2). Across the two studies, we found that automated updates tended to improve forecast accuracy relative to initial forecasts and were comparable to manual updates. Additionally, making rules improved the accuracy of manual updates. Crowd forecasts likewise benefitted from rule-based updates. However, when presented with the choice of whether to accept, reject or adjust an automatic forecast update, participants showed little ability to discriminate between automated updates that were harmful versus beneficial to forecast accuracy. Simple prospective rule-based tools are thus able to improve forecast accuracy by offering accurate and efficient updates, but ensuring forecasters make use of tools remains a challenge

Crowdsourcing for linguistic field research and e-learning

Crowdsourcing denotes the transfer of work commonly carried out by single humans to a large group of people. Nowadays, crowdsourcing is employed for many purposes, like people contributing their knowledge to Wikipedia, researchers predicting diseases from data on Twitter, or players solving protein folding problems in games. Still, there are areas for which the application of crowdsourcing has not yet been investigated thoroughly. This thesis examines crowdsourcing for two such areas: for empirical research in sciences oriented on humans -focusing on linguistic field research- and for e-learning. Sciences oriented on humans -like linguistics, sociology, or art history- depend on empirical research. For example, in traditional linguistic field research researchers ask questions and fill in forms. Such methods are time-consuming, costly, and not free of biases. This thesis proposes the application of crowdsourcing techniques to overcome these disadvantages and to support empirical research in getting more efficient. Therefore, the concept of a generic market for trading with symbolic goods and speculating on their characteristics in a playful manner, called Agora is introduced. Agora aims to be an "operating system" for social media applications gathering data. Furthermore, the Web-based crowdsourcing platform metropolitalia has been established for hosting two social media applications based upon Agora: Mercato Linguistico and Poker Parole. These applications have been conceived as part of this thesis for gathering complementary data and meta-data on Italian language varieties. Mercato Linguistico incites players to express their own knowledge or beliefs, Poker Parole incites players to make conjectures on the contributions of others. Thereby the primary meta-data collected with Mercato Linguistico are enriched with secondary, reflexive meta-data from Poker Parole, which are needed for studies on the perception of languages. An evaluation of the data gathered on metropolitalia exhibits the viability of the market-based approach of Agora and highlights its strengths. E-learning is concerned with the use of digital technology for learning, nowadays especially via the Internet. This thesis investigates how e-learning applications can support students with association-based learning and lecturers with teaching. For that, a game-like e-learning tool named Termina is proposed in this thesis. From the data collected with Termina association maps are constructed. An association map is a simplified version of a concept map, in which concepts are represented as rectangles and relationships between concepts as links. They constitute an abstract comprehension of a topic. Students profit from the association maps' availability, learn from other participating students, and can track their own learning progress. Lecturers gain insights into the knowledge and into potential misunderstandings of their students. An evaluation of Termina and the collected data along a university course exhibits Termina's usefulness for both students and lecturers. The main contributions of this thesis are (1) a literature review over collective intelligence, crowdsourcing, and related fields, (2) a model of a generic market for gathering data for empirical research efficiently, (3) two applications based on this model and results of an evaluation of the data gathered with them, (4) the game-like e-learning tool Termina together with insights from its evaluation, and (5) a generic software architecture for all aforementioned applications.Crowdsourcing bezeichnet die Auslagerung von Arbeit an eine Gruppe von Menschen zur Lösung eines Problems. Heutzutage wird Crowdsourcing für viele Zwecke verwendet, zum Beispiel tragen Leute ihr Wissen zu Wikipedia bei, Wissenschaftler sagen Krankheiten anhand von Twitter-Daten vorher oder Spieler lösen Proteinfaltungsprobleme in Spielen. Es gibt dennoch Gebiete, für die der Einsatz von Crowdsourcing noch nicht gründlich untersucht wurde. Diese Arbeit untersucht Crowdsourcing für zwei solche Gebiete: für empirische Forschung in auf den Menschen bezogenen Wissenschaften mit Fokus auf linguistischer Feldforschung sowie für E-Learning. Auf den Menschen bezogene Wissenschaften wie Linguistik, Soziologie oder Kunstgeschichte beruhen auf empirischer Forschung. In traditioneller linguistischer Feldforschung zum Beispiel stellen Wissenschaftler Fragen und füllen Fragebögen aus. Solche Methoden sind zeitaufwändig, teuer und nicht unbefangen. Diese Arbeit schlägt vor, Crowdsourcing-Techniken anzuwenden, um diese Nachteile zu überwinden und um empirische Forschung effizienter zu gestalten. Dazu wird das Konzept eines generischen Marktes namens Agora für den Handel mit symbolischen Gütern und für die Spekulation über deren Charakteristika eingeführt. Agora ist ein generisches "Betriebssystem" für Social Media Anwendungen. Außerdem wurde die Internet-basierte Crowdsourcing-Plattform metropolitalia eingerichtet, um zwei dieser Social Media Anwendungen, die auf Agora basieren, bereitzustellen: Mercato Linguistico und Poker Parole. Diese Anwendungen wurden als Teil dieser Arbeit entwickelt, um komplementäre Daten und Metadaten über italienische Sprachvarietäten zu sammeln. Mercato Linguistico regt Spieler dazu an, ihr eigenes Wissen und ihre Überzeugungen auszudrücken. Poker Parole regt Spieler dazu an, Vermutungen über die Beiträge anderer Spieler anzustellen. Damit werden die mit Mercato Linguistico gesammelten primären Metadaten mit reflexiven sekundären Metadaten aus Poker Parole, die für Studien über die Wahrnehmung von Sprachen notwendig sind, bereichert. Eine Auswertung der auf metropolitalia gesammelten Daten zeigt die Zweckmäßigkeit des marktbasierten Ansatzes von Agora und unterstreicht dessen Stärken. E-Learning befasst sich mit der Verwendung von digitalen Technologien für das Lernen, heutzutage vor allem über das Internet. Diese Arbeit untersucht, wie E-Learning-Anwendungen Studenten bei assoziationsbasiertem Lernen und Dozenten bei der Lehre unterstützen können. Dafür wird eine Spiel-ähnliche Anwendung namens Termina in dieser Arbeit eingeführt. Mit den über Termina gesammelten Daten werden Association-Maps konstruiert. Eine Association-Map ist eine vereinfachte Variante einer Concept-Map, in der Begriffe als Rechtecke und Beziehungen zwischen Begriffen als Verbindungslinien dargestellt werden. Sie stellen eine abstrakte Zusammenfassung eines Themas dar. Studenten profitieren von der Verfügbarkeit der Association-Maps, lernen von anderen Studenten und können ihren eigenen Lernprozess verfolgen. Dozenten bekommen Einblicke in den Wissensstand und in eventuelle Missverständnisse ihrer Studenten. Eine Evaluation von Termina und der damit gesammelten Daten während eines Universitätskurses bestätigt, dass Termina sowohl für Studenten als auch für Dozenten hilfreich ist. Die Kernbeiträge dieser Arbeit sind (1) eine Literaturrecherche über kollektive Intelligenz, Crowdsourcing und verwandte Gebiete, (2) ein Modell eines generischen Marktes zur effizienten Sammlung von Daten für empirische Forschung, (3) zwei auf diesem Modell basierende Anwendungen und Ergebnisse deren Evaluation, (4) die Spiel-ähnliche E-Learning-Anwendung Termina zusammen mit Einblicken aus dessen Evaluation und (5) eine generische Softwarearchitektur für alle vorgenannten Anwendungen

Engineering Delphi-Markets for Crowd-based Prediction - The FAZ.NET-Orakel and other Cases

Reliable forecasting is a key success factor of most organizations and companies. Where historical data is not available, the forecasts address questions in the far future, information is dispersed regarding location and form, or conflicting goals or values have to be considered, judgmental forecasting methods based on experts or the crowd are typically applied. However, several judgmental forecasting methods exist and each method has some individual weaknesses. Delphi-Markets are an integrated approach of prediction markets and Real-Time Delphi studies. Depending on their implementation, they allow to combine several properties of both approaches in order to overcome individual weaknesses. Three different ways to integrate the method are presented and discussed in this work. In order to better understand challenges and potentials of Delphi-Markets, the FAZ.NET-Orakel was instantiated and made publicly available for evaluation and improvement of an exemplary Delphi-Market under real-world conditions. In this context, four proposed improvements for the integrated approach were evaluated in four research projects. These projects correspond to the four sources of forecasting error according to the Judgmental Forecasting Improvement Model, introduced and derived in this dissertation as well. On the one hand, these improvements deal with common problems of prediction markets: Cognitive errors, such as partition dependence, and motivational errors, such as manipulation and fraud. On the other hand, these include common problems of Real-Time Delphi studies: The selection of experts for Delphi studies and retention during the surveys. As contributions to the overall IS research derived from the examinations of the Delphi-Markets and this dissertation, design principles for two extensions (social Real-Time Delphi and a crowd-based approach for manipulation and fraud detection) are formulated, implemented, tested, and suggested for application. Further, the role of complexity and expertise in the occurrence of the partition dependence bias is examined and a selection approach for experts for Delphi studies based on trading data is suggested and evaluated