FORETELL: Aggregating Distributed, Heterogeneous Information from Diverse Sources Using Market-based Techniques

Predicting the outcome of uncertain events that will happen in the future is a frequently indulged task by humans while making critical decisions. The process underlying this prediction and decision making is called information aggregation, which deals with collating the opinions of different people, over time, about the future event’s possible outcome. The information aggregation problem is non-trivial as the information related to future events is distributed spatially and temporally, the information gets changed dynamically as related events happen, and, finally, people’s opinions about events’ outcomes depends on the information they have access to and the mechanism they use to form opinions from that information. This thesis addresses the problem of distributed information aggregation by building computational models and algorithms for different aspects of information aggregation so that the most likely outcome of future events can be predicted with utmost accuracy. We have employed a commonly used market-based framework called a prediction market to formally analyze the process of information aggregation. The behavior of humans performing information aggregation within a prediction market is implemented using software agents which employ sophisticated algorithms to perform complex calculations on behalf of the humans, to aggregate information efficiently. We have considered five different yet crucial problems related to information aggregation, which include: (i) the effect of variations in the parameters of the information being aggregated, such as its reliability, availability, accessibility, etc., on the predicted outcome of the event, (ii) improving the prediction accuracy by having each human (software-agent) build a more accurate model of other humans’ behavior in the prediction market, (iii) identifying how various market parameters effect its dynamics and accuracy, (iv) applying information aggregation to the domain of distributed sensor information fusion, and, (v) aggregating information on an event while considering dissimilar, but closely-related events in different prediction markets. We have verified all of our proposed techniques through analytical results and experiments while using commercially available data from real prediction markets within a simulated, multi-agent based prediction market. Our results show that our proposed techniques for information aggregation perform more efficiently or comparably with existing techniques for information aggregation using prediction markets

Open Models of Decision Support Towards a Framework

Aquesta tesi presenta un marc per als models oberts de suport a les decisions en les organitzacions. El treball es vehicula a través d’un compendi d’articles on s’analitzen els fluxos d’entrada i de sortida de coneixement en les organitzacions, així como les tecnologies existents de suport a les decisions. Es presenten els factors subjacents que impulsen nous models per a formes obertes de suport a la decisió. La tesis presenta un estudi de les distintes tipologies de models de suport a les decisions tenint en compte diferents tipus d’organitzacions. En el primer estudi, paper#, es presenta l’evolució de les tecnologies de suport a les decisions i l’avançament de les noves tecnologies per als models oberts. Aquest estudi proporciona una visió des d’una perspectiva evolutiva de la relació entre el coneixement expert i la seva utilització en les tecnologies de suport a les decisions. La investigació revela l’entorn canviant que la tecnologia ofereix a l’hora de adquirir coneixement per a la presa de decisions i obre horitzons sobre el nou paper que els experts tenen en aquests entorns. Es suggereix que un canvi significatiu en la presa de decisions es basa en el desafiament entre el paper tradicional dels experts i no experts. Per últim, aquest treball explora les oportunitats d’integració de la intel•ligència artificial en la tecnologia de suport a les decisions i quins beneficis addicionals poden aportar les eines d’ intel•ligència col•lectiva en la presa de decisions. El segon estudi, paper#2, investiga sobre la tipologia anomenada "agregada" dins del marc d’entorns oberts per al suport a la presa de decisions. S’utilitza un problema de predicció com a fil conductor per a posar en relleu la complexitat de la previsió de la demanda dins de la industria del cinema. S’analitza com es pot utilitzar la tecnologia per a millorar l’eficàcia en les decisions. La investigació compara dues tecnologies de suport a les decisions: sistemes experts i eines d’intel•ligència col•lectiva, i il•lustra com l’industria del cinema utilitza cada una d’aquestes tecnologies en la previsió dels ingressos de taquilla. Per últim, aquest article explora els beneficis de l’ integració d’aquestes tecnologies de suport per a l’obtenció de prediccions més precises. El tercer estudi, article#3, presenta un estudi longitudinal durant un període de 10 anys que utilitza IBM “Innovation Jams” como un context per a la col•laboració a gran escala dins de la tipologia anomenada "plataforma". Aquest article investiga el paper de les “Innovation Jams”, en el canvi organitzacional i com IBM es compromet amb un nou model d’innovació en les organitzacions. En ell es descriuen les “Innovation Jams”, que han impulsat la innovació i consolidat la pràctica de la innovació oberta en IBM. En aquest article s’utilitza el gènere musical d’una "jamband" com una metàfora per a descriure el desenvolupament emergent i l’ús de les “Innovation Jams”, com una manera d’entendre el canvi organitzatiu. Aquest estudi longitudinal ofereix una visió actualitzada de la recerca en “Innovation Jams”, mostrant com han evolucionat des d’un concepte, a una eina de gestió i finalment a un servei. L’article conclou amb una discussió sobre les implicacions dels resultats i com aquests permeten teoritzar sobre nous models d’ innovació i el canvi en les organitzacions. La recerca duta a terme en aquesta tesi ofereix un marc per als models oberts de suport a la decisió, i suggereix que, les fonts internes i externes de coneixement poden ser utilitzades, més enllà de la innovació del producte o serveis, per a la presa de decisions amb el suport de tecnologies emergents. Les contribucions teòriques d’aquesta tesi sostenen que les organitzacions ja no poden confiar en la tecnologia de suport a les decisions que únicament es centren en la reducció de la frontera entre els aspectes racionals i no racionals de la conducta social humana, sinó que pel contrari, han de considerar la xarxa dinàmica de la organització per al suport a la decisió. D’altra banda, les implicacions pràctiques d’aquesta tesi animen les organitzacions a pensar estratègicament sobre com les tecnologies emergents poden ajudar en la presa de decisions i també com els models de decisió resultants poden ser utilitzats per a navegar per l’entorn complex existent, i, a la vegada, forjar vincles més forts amb els clients, proveïdors i la xarxa de l’organització.Esta tesis presenta un marco para modelos abiertos de soporte a las decisiones en las organizaciones. El trabajo se vehicula a través de un compendio de artículos dónde se analizan los flujos de entrada y salida de conocimiento en las organizaciones, así como las tecnologías existentes de soporte a las decisiones. Se presentan los factores subyacentes que impulsan nuevos modelos para formas abiertas de soporte a la decisión. La tesis presenta un estudio de las distintas tipologías de modelos de soporte a las decisiones teniendo en cuenta distintos tipos de organizaciones. En el primer estudio paper#1 se presenta la evolución de las tecnologías de apoyo a las decisiones y el avance de las nuevas tecnologías para los modelos abiertos. Este estudio proporciona una visión desde una perspectiva evolutiva de la relación entre conocimiento experto y su utilización en las tecnologías de soporte a las decisiones. La investigación revela el entorno cambiante que la tecnología ofrece a la hora de adquirir conocimiento para la toma de decisiones y abre horizontes sobre el nuevo papel que los expertos tienen en estos entornos. Se sugiere que un cambio significativo en la toma de decisiones se basa en el desafío entre el papel tradicional de los expertos y no expertos. Por último, este trabajo explora las oportunidades de integración de la inteligencia artificial en la tecnología de soporte de decisiones y que beneficios adicionales pueden aportar las herramientas de inteligencia colectiva en la toma de decisiones. El segundo estudio, paper#2, investiga sobre la tipología llamada "agregada" dentro del marco de entornos abiertos para el soporte a la toma de decisiones. Se utiliza un problema de predicción como hilo conductor para poner en relieve la complejidad de la previsión de la demanda dentro de la industria del cine. Se analiza cómo se puede utilizar la tecnología para mejorar la eficacia en las decisiones. La investigación compara dos tecnologías de soporte a las decisiones: sistemas expertos y herramientas de inteligencia colectiva, e ilustra cómo la industria del cine utiliza cada una de estas tecnologías en la previsión de los ingresos de taquilla. Por último, este artículo explora los beneficios de la integración de estas tecnologías de apoyo para la obtención de predicciones más precisas. El tercer estudio, artículo #3, presenta un estudio longitudinal durante un período de 10 años que utiliza IBM “Innovation Jams”, como un contexto para la colaboración a gran escala dentro de la tipología llamada "plataforma". Este artículo investiga el papel de las “Innovation Jams”, en el cambio organizacional y como IBM se compromete con un nuevo modelo de innovación de la organización. En él se describen las “Innovation Jams”, que han impulsado la innovación y consolidado la práctica de la innovación abierta en IBM. En este artículo se utiliza el género musical de una "jamband" como una metáfora para describir el desarrollo emergente y el uso de las “Innovation Jams”, como una manera de entender el cambio organizativo. Este estudio longitudinal ofrece una visión actualizada de la investigación en “Innovation Jams”, mostrando cómo han evolucionado desde un concepto, a una herramienta de gestión y finalmente a un servicio. El artículo concluye con una discusión sobre las implicaciones de los resultados y como ellos permiten teorizar sobre nuevos modelos de innovación y el cambio en las organizaciones. La investigación llevada a cabo en esta tesis ofrece un marco para los modelos abiertos de apoyo a la decisión, y sugiere que el uso de fuentes internas y externas de conocimiento pueden ser utilizadas más allá de la innovación del producto o servicio para la toma de decisiones con el soporte de tecnologías emergentes. Las contribuciones teóricas de esta tesis sostienen que las organizaciones ya no pueden confiar en la tecnología de apoyo a las decisiones que únicamente se centran en la reducción de la frontera entre los aspectos racionales y no racionales de la conducta social humana, sino por el contrario, deben considerar la red dinámica de la organización para el apoyo a la decisión. Por otra parte, las implicaciones prácticas de esta tesis alienta a las organizaciones a pensar estratégicamente acerca de cómo las tecnologías emergentes pueden ayudar a la toma de decisiones y también cómo los modelos de decisión resultantes pueden ser utilizados para navegar por el entorno complejo existente y, a su vez, forjar vínculos más fuertes con los clientes, proveedores y más amplios de la red de la organización.This thesis presents a framework for open models of decision support through a compendium of papers that links research on the inward and outward flows of knowledge to the organization and decision support technologies. The framework presents underlying factors driving new and more open models of decision support. A typology of decision support models is offered considering types of problems organizations and managers charged with decision-making face. Thesis essay #1 suggests a perspective of the changing landscape for decision support technology and the advancement of new technology for open models of decision support. This study provides insight from an evolutionary perspective of expertise that has shaped the field of decision support technologies. The investigation sets out to reveal the changing landscape of expertise in supporting decision-making using technology and sheds light on the new role that experts will play in organizational decision-making. It suggests that a significant change in how decision-making is being supported which challenge the traditional role of experts and non-experts. Finally, this paper explores opportunities for decision support technology integration and the added benefits artificial intelligence can bring to collective intelligence tools. Thesis essay #2 investigates the ‘aggregate’ typology within the open model decision support framework. A forecasting problem is used to highlight the complexity of demand forecasting in supply-chain management within the film industry and how technology is leveraged for effective supply-chain management decisions. The investigation compares two decision support technologies: expert systems and collective intelligence tools and illustrates how the film industry uses each in forecasting box-office revenue. Finally, this essay explores the combined benefits in integrating each support technology for more accurate forecasting. Thesis essay #3 is a longitudinal study over a 10 year period that uses IBM Innovation Jams as a context for large-scale collaboration within the ‘platform’ typology. This essay investigates the role of innovation jams on organizational change as IBM learned to engage with a new model of organizing innovation. It describes the role innovation jams have played in shaping the practice of open innovation at IBM. This essay uses the musical genre of a “jamband” as a metaphor to describe the emergent development and use of innovation jams as a way to understand organizational change. This longitudinal study brings innovation jam research up-to-date and presents innovation jams as they evolved from a concept, a management tool, and service. The essay concludes with a discussion on the implications of the findings for theorizing about new models of organizing innovation for organizational change. Research conducted in this thesis offers a framework of open models of decision support that suggests that the use of internal and external sources of knowledge can be leveraged beyond product or service innovation, to include decision-making supported by emerging technology. Theoretical contributions of this thesis argues that organizations can no longer rely on decision support technology that solely focus on bridging the boundary between rational and non-rational aspects of human social behavior but instead, must consider the larger dynamic organizational network for decision support. Moreover, practical implications of this thesis encourages organizations to think strategically about how emerging technology can support decision making and the resulting decision support models to navigate the complex environment they work in and in turn, to forge stronger links with customers, suppliers, and the wider organizational network

Coherent approximation of distributed expert assessments

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2011.Cataloged from PDF version of thesis.Includes bibliographical references (p. 157-168).Expert judgments of probability and expectation play an integral role in many systems. Financial markets, public policy, medical diagnostics and more rely on the ability of informed experts (both human and machine) to make educated assessments of the likelihood of various outcomes. Experts however are not immune to errors in judgment (due to bias, quantization effects, finite information or many other factors). One way to compensate for errors in individual judgments is to elicit estimates from multiple experts and then fuse the estimates together. If the experts act sufficiently independently to form their assessments, it is reasonable to assume that individual errors in judgment can be negated by pooling the experts' opinions. Determining when experts' opinions are in error is not always a simple matter. However, one common way in which experts' opinions may be seen to be in error is through inconsistency with the known underlying structure of the space of events. Not only is structure useful in identifying expert error, it should also be taken into account when designing algorithms to approximate or fuse conflicting expert assessments. This thesis generalizes previously proposed constrained optimization methods for fusing expert assessments of uncertain events and quantities. The major development consists of a set of information geometric tools for reconciling assessments that are inconsistent with the assumed structure of the space of events. This work was sponsored by the U.S. Air Force under Air Force Contract FA8721- 05-C-0002. Opinions, interpretations, conclusions, and recommendations are those of the authors and are not necessarily endorsed by the United States Government.by Peter B. Jones.Ph.D

Estimating credibility of science claims : analysis of forecasting data from metascience projects : a thesis presented in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Statistics at Massey University, Albany, New Zealand

The veracity of scientific claims is not always certain. In fact, sufficient claims have been proven incorrect that many scientists believe that science itself is facing a “replication crisis”. Large scale replication projects provided empirical evidence that only around 50% of published social and behavioral science findings are replicable. Multiple forecasting studies showed that the outcomes of replication projects could be predicted by crowdsourced human evaluators. The research presented in this thesis builds on previous forecasting studies, deriving new findings and exploring new scope and scale. The research is centered around the DARPA SCORE (Systematizing Confidence in Open Research and Evidence) programme, a project aimed at developing measures of credibility for social and behavioral science claims. As part of my contribution to SCORE, myself, along with a international collaboration, elicited forecasts from human experts via surveys and prediction markets to predict the replicability of 3000 claims. I also present research on other forecasting studies. In chapter 2, I pool data from previous studies to analyse the performance of prediction markets and surveys with higher statistical power. I confirm that prediction markets are better at forecasting replication outcomes than surveys. This study also demonstrates the relationship between p-values of original findings and replication outcomes. These findings are used to inform the experimental and statistical design to forecast the replicability of 3000 claims as part of the SCORE programme. A full description of the design including planned statistical analyses is included in chapter 3. Due to COVID-19 restrictions, our generated forecasts could not be validated through direct replication, experiments conducted by other teams within the SCORE collaboration, thereby preventing results being presented in this thesis. The completion of these replications is now scheduled for 2022, and the pre-analysis plan presented in Chapter 3 will provide the basis for the analysis of the resulting data. In chapter 4, an analysis of ‘meta’ forecasts, or forecasts regarding field wide replication rates and year specific replication rates, is presented. We presented and published community expectations that replication rates will differ by field and will increase over time. These forecasts serve as valuable insights into the academic community’s views of the replication crisis, including those research fields for which no large-scale replication studies have been undertaken yet. Once the full results from SCORE are available, there will be additional insights from validations of the community expectations. I also analyse forecaster’s ability to predict replications and effect sizes in Chapters 5 (Creative Destruction in Science) and 6 (A creative destruction approach to replication: Implicit work and sex morality across cultures). In these projects a ‘creative destruction’ approach to replication was used, where a claim is compared not only to the null hypothesis but to alternative contradictory claims. I conclude forecasters can predict the size and direction of effects. Chapter 7 examines the use of forecasting for scientific outcomes beyond replication. In the COVID-19 preprint forecasting project I find that forecasters can predict if a preprint will be published within one year, including the quality of the publishing journal. Forecasters can also predict the number of citations preprints will receive. This thesis demonstrates that information about scientific claims with respect to replicability is dispersed within scientific community. I have helped to develop methodologies and tools to efficiently elicit and aggregate forecasts. Forecasts about scientific outcomes can be used as guides to credibility, to gauge community expectations and to efficiently allocate sparse replication resources

Applications of agent architectures to decision support in distributed simulation and training systems

This work develops the approach and presents the results of a new model for applying intelligent agents to complex distributed interactive simulation for command and control. In the framework of tactical command, control communications, computers and intelligence (C4I), software agents provide a novel approach for efficient decision support and distributed interactive mission training. An agent-based architecture for decision support is designed, implemented and is applied in a distributed interactive simulation to significantly enhance the command and control training during simulated exercises. The architecture is based on monitoring, evaluation, and advice agents, which cooperate to provide alternatives to the dec ision-maker in a time and resource constrained environment. The architecture is implemented and tested within the context of an AWACS Weapons Director trainer tool. The foundation of the work required a wide range of preliminary research topics to be covered, including real-time systems, resource allocation, agent-based computing, decision support systems, and distributed interactive simulations. The major contribution of our work is the construction of a multi-agent architecture and its application to an operational decision support system for command and control interactive simulation. The architectural design for the multi-agent system was drafted in the first stage of the work. In the next stage rules of engagement, objective and cost functions were determined in the AWACS (Airforce command and control) decision support domain. Finally, the multi-agent architecture was implemented and evaluated inside a distributed interactive simulation test-bed for AWACS Vv\u27Ds. The evaluation process combined individual and team use of the decision support system to improve the performance results of WD trainees. The decision support system is designed and implemented a distributed architecture for performance-oriented management of software agents. The approach provides new agent interaction protocols and utilizes agent performance monitoring and remote synchronization mechanisms. This multi-agent architecture enables direct and indirect agent communication as well as dynamic hierarchical agent coordination. Inter-agent communications use predefined interfaces, protocols, and open channels with specified ontology and semantics. Services can be requested and responses with results received over such communication modes. Both traditional (functional) parameters and nonfunctional (e.g. QoS, deadline, etc.) requirements and captured in service requests

Engineering Delphi-Markets for Crowd-based Prediction - The FAZ.NET-Orakel and other Cases

Reliable forecasting is a key success factor of most organizations and companies. Where historical data is not available, the forecasts address questions in the far future, information is dispersed regarding location and form, or conflicting goals or values have to be considered, judgmental forecasting methods based on experts or the crowd are typically applied. However, several judgmental forecasting methods exist and each method has some individual weaknesses. Delphi-Markets are an integrated approach of prediction markets and Real-Time Delphi studies. Depending on their implementation, they allow to combine several properties of both approaches in order to overcome individual weaknesses. Three different ways to integrate the method are presented and discussed in this work. In order to better understand challenges and potentials of Delphi-Markets, the FAZ.NET-Orakel was instantiated and made publicly available for evaluation and improvement of an exemplary Delphi-Market under real-world conditions. In this context, four proposed improvements for the integrated approach were evaluated in four research projects. These projects correspond to the four sources of forecasting error according to the Judgmental Forecasting Improvement Model, introduced and derived in this dissertation as well. On the one hand, these improvements deal with common problems of prediction markets: Cognitive errors, such as partition dependence, and motivational errors, such as manipulation and fraud. On the other hand, these include common problems of Real-Time Delphi studies: The selection of experts for Delphi studies and retention during the surveys. As contributions to the overall IS research derived from the examinations of the Delphi-Markets and this dissertation, design principles for two extensions (social Real-Time Delphi and a crowd-based approach for manipulation and fraud detection) are formulated, implemented, tested, and suggested for application. Further, the role of complexity and expertise in the occurrence of the partition dependence bias is examined and a selection approach for experts for Delphi studies based on trading data is suggested and evaluated