Finite-time influence systems and the Wisdom of Crowd effect

Recent contributions have studied how an influence system may affect the wisdom of crowd phenomenon. In the so-called naive learning setting, a crowd of individuals holds opinions that are statistically independent estimates of an unknown parameter; the crowd is wise when the average opinion converges to the true parameter in the limit of infinitely many individuals. Unfortunately, even starting from wise initial opinions, a crowd subject to certain influence systems may lose its wisdom. It is of great interest to characterize when an influence system preserves the crowd wisdom effect. In this paper we introduce and characterize numerous wisdom preservation properties of the basic French-DeGroot influence system model. Instead of requiring complete convergence to consensus as in the previous naive learning model by Golub and Jackson, we study finite-time executions of the French-DeGroot influence process and establish in this novel context the notion of prominent families (as a group of individuals with outsize influence). Surprisingly, finite-time wisdom preservation of the influence system is strictly distinct from its infinite-time version. We provide a comprehensive treatment of various finite-time wisdom preservation notions, counterexamples to meaningful conjectures, and a complete characterization of equal-neighbor influence systems

Empirical Methodology for Crowdsourcing Ground Truth

The process of gathering ground truth data through human annotation is a major bottleneck in the use of information extraction methods for populating the Semantic Web. Crowdsourcing-based approaches are gaining popularity in the attempt to solve the issues related to volume of data and lack of annotators. Typically these practices use inter-annotator agreement as a measure of quality. However, in many domains, such as event detection, there is ambiguity in the data, as well as a multitude of perspectives of the information examples. We present an empirically derived methodology for efficiently gathering of ground truth data in a diverse set of use cases covering a variety of domains and annotation tasks. Central to our approach is the use of CrowdTruth metrics that capture inter-annotator disagreement. We show that measuring disagreement is essential for acquiring a high quality ground truth. We achieve this by comparing the quality of the data aggregated with CrowdTruth metrics with majority vote, over a set of diverse crowdsourcing tasks: Medical Relation Extraction, Twitter Event Identification, News Event Extraction and Sound Interpretation. We also show that an increased number of crowd workers leads to growth and stabilization in the quality of annotations, going against the usual practice of employing a small number of annotators.Comment: in publication at the Semantic Web Journa

Gradient descent for sparse rank-one matrix completion for crowd-sourced aggregation of sparsely interacting workers

We consider worker skill estimation for the singlecoin Dawid-Skene crowdsourcing model. In practice skill-estimation is challenging because worker assignments are sparse and irregular due to the arbitrary, and uncontrolled availability of workers. We formulate skill estimation as a rank-one correlation-matrix completion problem, where the observed components correspond to observed label correlation between workers. We show that the correlation matrix can be successfully recovered and skills identifiable if and only if the sampling matrix (observed components) is irreducible and aperiodic. We then propose an efficient gradient descent scheme and show that skill estimates converges to the desired global optima for such sampling matrices. Our proof is original and the results are surprising in light of the fact that even the weighted rank-one matrix factorization problem is NP hard in general. Next we derive sample complexity bounds for the noisy case in terms of spectral properties of the signless Laplacian of the sampling matrix. Our proposed scheme achieves state-of-art performance on a number of real-world datasets.Published versio

Challenges in Complex Systems Science

FuturICT foundations are social science, complex systems science, and ICT. The main concerns and challenges in the science of complex systems in the context of FuturICT are laid out in this paper with special emphasis on the Complex Systems route to Social Sciences. This include complex systems having: many heterogeneous interacting parts; multiple scales; complicated transition laws; unexpected or unpredicted emergence; sensitive dependence on initial conditions; path-dependent dynamics; networked hierarchical connectivities; interaction of autonomous agents; self-organisation; non-equilibrium dynamics; combinatorial explosion; adaptivity to changing environments; co-evolving subsystems; ill-defined boundaries; and multilevel dynamics. In this context, science is seen as the process of abstracting the dynamics of systems from data. This presents many challenges including: data gathering by large-scale experiment, participatory sensing and social computation, managing huge distributed dynamic and heterogeneous databases; moving from data to dynamical models, going beyond correlations to cause-effect relationships, understanding the relationship between simple and comprehensive models with appropriate choices of variables, ensemble modeling and data assimilation, modeling systems of systems of systems with many levels between micro and macro; and formulating new approaches to prediction, forecasting, and risk, especially in systems that can reflect on and change their behaviour in response to predictions, and systems whose apparently predictable behaviour is disrupted by apparently unpredictable rare or extreme events. These challenges are part of the FuturICT agenda

In crowdfunding we trust : a trust-building model in lending crowdfunding

Trust critically affects the perceived probability of receiving expected returns on investment. Crowdfunding differs in many ways from traditional forms of investing. We have to ask what builds trust in this particular context. Based on literature regarding the formation of initial trust, we developed a model to explain which factors lead to crowdfunders’ trust in a crowdfunding project. We tested it on data collected from actual investors in a real project on a crowdlending platform. Our results show that trust in the crowdfunding platform and the information quality are more important factors of project trust than trust in the creator