Balancing Human and Machine Contributions in Human Computation Systems

Many interesting and successful human computation systems leverage the complementary computational strengths of both humans and machines to solve these problems. In this chapter, we examine Human Computation as a type of Human-Computer Collaboration—collaboration involving at least one human and at least one computational agent. We discuss recent advances in the open area of function allocation, and explore how to balance the contributions of humans and machines in computational systems. We then explore how human-computer collaborative strategies can be used to solve problems that are difficult or computationally infeasible for computers or humans alone

Diverse strategic identities induce dynamical states in evolutionary games

Evolutionary games provide the theoretical backbone for many aspects of our social life: from cooperation to crime, from climate inaction to imperfect vaccination and epidemic spreading, from antibiotics overuse to biodiversity preservation. An important, and so far overlooked, aspect of reality is the diverse strategic identities of individuals. While applying the same strategy to all interaction partners may be an acceptable assumption for simpler forms of life, this fails to account} for the behavior of more complex living beings. For instance, we humans act differently around different people. Here we show that allowing individuals to adopt different strategies with different partners yields a very rich evolutionary dynamics, including time-dependent coexistence of cooperation and defection, system-wide shifts in the dominant strategy, and maturation in individual choices. Our results are robust to variations in network type and size, and strategy updating rules. Accounting for diverse strategic identities thus has far-reaching implications in the mathematical modeling of social games.Comment: 9 pages, 4 figure

Don’t Look Here: Off-Limits Words Bias Play in the ESP Game

Social computation is a paradigm in which a software application supports social interaction, but whose real purpose is the data trail that is left by that interaction, such as tags, recommendations, and so on. We explore a possible imperfection in the data that is generated by these games. Specifically, we investigate whether the data generated by previous participants influences the data that is generated by subsequent participants. We investigate this in the context of the ESP game. A feature of the ESP game is that words that have been generated by previous players are off-limits to subsequent players. The idea of this feature is to ensure that the system accumulates a variety of different words for each image. We consider the possibility that ironically, players are actually biased to suggest words that are related to the taboo words themselves. Based on anchoring and priming theories, we predict that the words players suggest will be related to the taboo words, and that this phenomenon will limit the variety of words that are collected for a given image. An empirical experiment confirms these predictions. This effect threatens to limit the potential value of socially generated information in many applications including recommender systems and ESP-like tagging systems, where later contributors are exposed to the inputs provided by earlier contributors

Toward Theoretical Techniques for Measuring the Use of Human Effort in Visual Analytic Systems

Visual analytic systems have long relied on user studies and standard datasets to demonstrate advances to the state of the art, as well as to illustrate the efficiency of solutions to domain-specific challenges. This approach has enabled some important comparisons between systems, but unfortunately the narrow scope required to facilitate these comparisons has prevented many of these lessons from being generalized to new areas. At the same time, advanced visual analytic systems have made increasing use of human-machine collaboration to solve problems not tractable by machine computation alone. To continue to make progress in modeling user tasks in these hybrid visual analytic systems, we must strive to gain insight into what makes certain tasks more complex than others. This will require the development of mechanisms for describing the balance to be struck between machine and human strengths with respect to analytical tasks and workload. In this paper, we argue for the necessity of theoretical tools for reasoning about such balance in visual analytic systems and demonstrate the utility of the Human Oracle Model for this purpose in the context of sensemaking in visual analytics. Additionally, we make use of the Human Oracle Model to guide the development of a new system through a case study in the domain of cybersecurity

A survey on the design of gamified systems for energy and water sustainability

Abstract In a world affected by the constant growth and concentration of the population in urban areas, the problem of preserving natural resources has become a priority. A promising approach to resource conservation is demand management, i.e., the ability to positively influence the behaviour of the population towards more sustainable consumption. Information and Communication Technology (ICT) tools have shown a great potential in influencing consumers’ behaviour, which could be exploited for the common good. However, the design of so-called persuasive systems for environmental purposes is a challenging task, because it cannot be based solely on the utilitarian motivation of users, but must be able to trigger a broader range of engagement factors deeply rooted in the human psychology. In this paper, we review the main design principles and models at the base of a class of persuasive system that exploits gamification and Games with a Purpose (GWAPs) to engage users towards sustainability; we identify the most commonly used incentive mechanisms for inducing behavioural changes; and present a selection of gamified systems for energy and water conservation. From such a survey, we distill design guidelines to be applied in the design of demand management socio-technical systems

Crowdsourcing decision support: frugal human computation for efficient decision input acquisition

When faced with data-intensive decision problems, individuals, businesses, and governmental decision-makers must balance trade-offs between optimality and the high cost of conducting a thorough decision process. The unprecedented availability of information online has created opportunities to make well-informed, near-optimal decisions more efficiently. A key challenge that remains is the difficulty of efficiently gathering the requisite details in a form suitable for making the decision. Human computation and social media have opened new avenues for gathering relevant information or opinions in support of a decision-making process. It is now possible to coordinate paid web workers from online labor markets such as Amazon Mechanical Turk and others in a distributed search party for the needed information. However, the strategies that individuals employ when confronted with too much information--satisficing, information foraging, etc.--are more difficult to apply with a large, distributed group. Consequently, current distributed approaches are inherently wasteful of human time and effort. This dissertation offers a method for coordinating workers to efficiently enter the inputs for spreadsheet decision models. As a basis for developing and understanding the idea, I developed AskSheet, a system that uses decision models represented as spreadsheets. The user provides a spreadsheet model of a decision, the formulas of which are analyzed to calculate the value of information for each of the decision inputs. With that, it is able to prioritize the inputs and make the decision input acquisition process more frugal. In doing so, it trades machine capacity for analyzing the model for a reduction in the cost and burden to the humans providing the needed information

Designing for quality in real-world mobile crowdsourcing systems

PhD ThesisCrowdsourcing has emerged as a popular means to collect and analyse data on a scale for problems that require human intelligence to resolve. Its prompt response and low cost have made it attractive to businesses and academic institutions. In response, various online crowdsourcing platforms, such as Amazon MTurk, Figure Eight and Prolific have successfully emerged to facilitate the entire crowdsourcing process. However, the quality of results has been a major concern in crowdsourcing literature. Previous work has identified various key factors that contribute to issues of quality and need to be addressed in order to produce high quality results. Crowd tasks design, in particular, is a major key factor that impacts the efficiency and effectiveness of crowd workers as well as the entire crowdsourcing process. This research investigates crowdsourcing task designs to collect and analyse two distinct types of data, and examines the value of creating high-quality crowdwork activities on new crowdsource enabled systems for end-users. The main contribution of this research includes 1) a set of guidelines for designing crowdsourcing tasks that support quality collection, analysis and translation of speech and eye tracking data in real-world scenarios; and 2) Crowdsourcing applications that capture real-world data and coordinate the entire crowdsourcing process to analyse and feed quality results back. Furthermore, this research proposes a new quality control method based on workers trust and self-verification. To achieve this, the research follows the case study approach with a focus on two real-world data collection and analysis case studies. The first case study, Speeching, explores real-world speech data collection, analysis, and feedback for people with speech disorder, particularly with Parkinson’s. The second case study, CrowdEyes, examines the development and use of a hybrid system combined of crowdsourcing and low-cost DIY mobile eye trackers for real-world visual data collection, analysis, and feedback. Both case studies have established the capability of crowdsourcing to obtain high quality responses comparable to that of an expert. The Speeching app, and the provision of feedback in particular were well perceived by the participants. This opens up new opportunities in digital health and wellbeing. Besides, the proposed crowd-powered eye tracker is fully functional under real-world settings. The results showed how this approach outperforms all current state-of-the-art algorithms under all conditions, which opens up the technology for wide variety of eye tracking applications in real-world settings

Crowdsource Annotation and Automatic Reconstruction of Online Discussion Threads

Modern communication relies on electronic messages organized in the form of discussion threads. Emails, IMs, SMS, website comments, and forums are all composed of threads, which consist of individual user messages connected by metadata and discourse coherence to messages from other users. Threads are used to display user messages effectively in a GUI such as an email client, providing a background context for understanding a single message. Many messages are meaningless without the context provided by their thread. However, a number of factors may result in missing thread structure, ranging from user mistake (replying to the wrong message), to missing metadata (some email clients do not produce/save headers that fully encapsulate thread structure; and, conversion of archived threads from over repository to another may also result in lost metadata), to covert use (users may avoid metadata to render discussions difficult for third parties to understand). In the field of security, law enforcement agencies may obtain vast collections of discussion turns that require automatic thread reconstruction to understand. For example, the Enron Email Corpus, obtained by the Federal Energy Regulatory Commission during its investigation of the Enron Corporation, has no inherent thread structure. In this thesis, we will use natural language processing approaches to reconstruct threads from message content. Reconstruction based on message content sidesteps the problem of missing metadata, permitting post hoc reorganization and discussion understanding. We will investigate corpora of email threads and Wikipedia discussions. However, there is a scarcity of annotated corpora for this task. For example, the Enron Emails Corpus contains no inherent thread structure. Therefore, we also investigate issues faced when creating crowdsourced datasets and learning statistical models of them. Several of our findings are applicable for other natural language machine classification tasks, beyond thread reconstruction. We will divide our investigation of discussion thread reconstruction into two parts. First, we explore techniques needed to create a corpus for our thread reconstruction research. Like other NLP pairwise classification tasks such as Wikipedia discussion turn/edit alignment and sentence pair text similarity rating, email thread disentanglement is a heavily class-imbalanced problem, and although the advent of crowdsourcing has reduced annotation costs, the common practice of crowdsourcing redundancy is too expensive for class-imbalanced tasks. As the first contribution of this thesis, we evaluate alternative strategies for reducing crowdsourcing annotation redundancy for class-imbalanced NLP tasks. We also examine techniques to learn the best machine classifier from our crowdsourced labels. In order to reduce noise in training data, most natural language crowdsourcing annotation tasks gather redundant labels and aggregate them into an integrated label, which is provided to the classifier. However, aggregation discards potentially useful information from linguistically ambiguous instances. For the second contribution of this thesis, we show that, for four of five natural language tasks, filtering of the training dataset based on crowdsource annotation item agreement improves task performance, while soft labeling based on crowdsource annotations does not improve task performance. Second, we investigate thread reconstruction as divided into the tasks of thread disentanglement and adjacency recognition. We present the Enron Threads Corpus, a newly-extracted corpus of 70,178 multi-email threads with emails from the Enron Email Corpus. In the original Enron Emails Corpus, emails are not sorted by thread. To disentangle these threads, and as the third contribution of this thesis, we perform pairwise classification, using text similarity measures on non-quoted texts in emails. We show that i) content text similarity metrics outperform style and structure text similarity metrics in both a class-balanced and class-imbalanced setting, and ii) although feature performance is dependent on the semantic similarity of the corpus, content features are still effective even when controlling for semantic similarity. To reconstruct threads, it is also necessary to identify adjacency relations among pairs. For the forum of Wikipedia discussions, metadata is not available, and dialogue act typologies, helpful for other domains, are inapplicable. As our fourth contribution, via our experiments, we show that adjacency pair recognition can be performed using lexical pair features, without a dialogue act typology or metadata, and that this is robust to controlling for topic bias of the discussions. Yet, lexical pair features do not effectively model the lexical semantic relations between adjacency pairs. To model lexical semantic relations, and as our fifth contribution, we perform adjacency recognition using extracted keyphrases enhanced with semantically related terms. While this technique outperforms a most frequent class baseline, it fails to outperform lexical pair features or tf-idf weighted cosine similarity. Our investigation shows that this is the result of poor word sense disambiguation and poor keyphrase extraction causing spurious false positive semantic connections. In concluding this thesis, we also reflect on open issues and unanswered questions remaining after our research contributions, discuss applications for thread reconstruction, and suggest some directions for future work

Mathematical modeling of social games

Human computation is a technique that makes use of human abilities for computation to solve problems. Social games use the power of the Internet game players to solve human computation problems. In previous works, many social games were proposed and were quite successful, but no formal framework exists for designing social games in general. A formal framework is important because it lists out the design elements of a social game, the characteristics of a human computation problem, and their relationships. With a formal framework, it simplifies the way to design a social game for a specific problem. In this paper, our contributions are: (1) formulate a formal model on social games, (2) analyze the framework and derive some interesting properties based on model’s interactions, (3) illustrate how some current social games can be realized with the proposed formal model, and (4) describe how to design a social game for solving a specific problem with the use of the proposed formal model. This paper presents a set of design guidelines derived from the formal model and demonstrates that the model can help to design a social game for solving a specific problem in a formal and structural way