Optimization techniques for human computation-enabled data processing systems

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2012.Cataloged from PDF version of thesis.Includes bibliographical references (p. 119-124).Crowdsourced labor markets make it possible to recruit large numbers of people to complete small tasks that are difficult to automate on computers. These marketplaces are increasingly widely used, with projections of over $1 billion being transferred between crowd employers and crowd workers by the end of 2012. While crowdsourcing enables forms of computation that artificial intelligence has not yet achieved, it also presents crowd workflow designers with a series of challenges including describing tasks, pricing tasks, identifying and rewarding worker quality, dealing with incorrect responses, and integrating human computation into traditional programming frameworks. In this dissertation, we explore the systems-building, operator design, and optimization challenges involved in building a crowd-powered workflow management system. We describe a system called Qurk that utilizes techniques from databases such as declarative workflow definition, high-latency workflow execution, and query optimization to aid crowd-powered workflow developers. We study how crowdsourcing can enhance the capabilities of traditional databases by evaluating how to implement basic database operators such as sorts and joins on datasets that could not have been processed using traditional computation frameworks. Finally, we explore the symbiotic relationship between the crowd and query optimization, enlisting crowd workers to perform selectivity estimation, a key component in optimizing complex crowd-powered workflows.by Adam Marcus.Ph.D

Crowd-supervised training of spoken language systems

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2012.Cataloged from PDF version of thesis.Includes bibliographical references (p. 155-166).Spoken language systems are often deployed with static speech recognizers. Only rarely are parameters in the underlying language, lexical, or acoustic models updated on-the-fly. In the few instances where parameters are learned in an online fashion, developers traditionally resort to unsupervised training techniques, which are known to be inferior to their supervised counterparts. These realities make the development of spoken language interfaces a difficult and somewhat ad-hoc engineering task, since models for each new domain must be built from scratch or adapted from a previous domain. This thesis explores an alternative approach that makes use of human computation to provide crowd-supervised training for spoken language systems. We explore human-in-the-loop algorithms that leverage the collective intelligence of crowds of non-expert individuals to provide valuable training data at a very low cost for actively deployed spoken language systems. We also show that in some domains the crowd can be incentivized to provide training data for free, as a byproduct of interacting with the system itself. Through the automation of crowdsourcing tasks, we construct and demonstrate organic spoken language systems that grow and improve without the aid of an expert. Techniques that rely on collecting data remotely from non-expert users, however, are subject to the problem of noise. This noise can sometimes be heard in audio collected from poor microphones or muddled acoustic environments. Alternatively, noise can take the form of corrupt data from a worker trying to game the system - for example, a paid worker tasked with transcribing audio may leave transcripts blank in hopes of receiving a speedy payment. We develop strategies to mitigate the effects of noise in crowd-collected data and analyze their efficacy. This research spans a number of different application domains of widely-deployed spoken language interfaces, but maintains the common thread of improving the speech recognizer's underlying models with crowd-supervised training algorithms. We experiment with three central components of a speech recognizer: the language model, the lexicon, and the acoustic model. For each component, we demonstrate the utility of a crowd-supervised training framework. For the language model and lexicon, we explicitly show that this framework can be used hands-free, in two organic spoken language systems.by Ian C. McGraw.Ph.D

Crowd-powered systems

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2012.Cataloged from PDF version of thesis.Includes bibliographical references (p. 217-237).Crowd-powered systems combine computation with human intelligence, drawn from large groups of people connecting and coordinating online. These hybrid systems enable applications and experiences that neither crowds nor computation could support alone. Unfortunately, crowd work is error-prone and slow, making it difficult to incorporate crowds as first-order building blocks in software systems. I introduce computational techniques that decompose complex tasks into simpler, verifiable steps to improve quality, and optimize work to return results in seconds. These techniques develop crowdsourcing as a platform so that it is reliable and responsive enough to be used in interactive systems. This thesis develops these ideas through a series of crowd-powered systems. The first, Soylent, is a word processor that uses paid micro-contributions to aid writing tasks such as text shortening and proofreading. Using Soylent is like having access to an entire editorial staff as you write. The second system, Adrenaline, is a camera that uses crowds to help amateur photographers capture the exact right moment for a photo. It finds the best smile and catches subjects in mid-air jumps, all in realtime. Moving beyond generic knowledge and paid crowds, I introduce techniques to motivate a social network that has specific expertise, and techniques to data mine crowd activity traces in support of a large number of uncommon user goals. These systems point to a future where social and crowd intelligence are central elements of interaction, software, and computation.by Michael Scott Bernstein.Ph.D

VELUM: A 3D Puzzle/Exploration Game Designed Using Crowdsourced AI Facial Analysis

Velum is a first-person 3D puzzle/exploration game set in a timeless version of the Boston Public Garden. The project’s narrative framework and aesthetics are based on one of the Garden’s most prominent features, the Ether Monument, which commemorates the 1846 discovery of diethyl ether’s effectiveness as a medical anesthetic. A sequence of nine abstract challenges is rewarded by a progressive revelation of the player’s mysterious identity and purpose. The puzzle design was informed by the use of crowdsourced playtesting involving 300+ volunteers, combining standard data telemetry with AI-based facial image analysis capable of mapping player emotions to gameplay events

A Computational Neuroscience Approach to Higher-Order Texture Perception

Natural images contain large amounts of structural information characterised by higher-order spatial correlations. Neurons have limited capacities, so the visual system must filter out non-salient information, but retain that which is behaviourally relevant. Previous research has concentrated on two-point correlations; there has been less research into higher-order correlations, although the visual system is sensitive to them. Isotrigon textures can be used for this purpose. Their salient structure is exclusively due to fourth- and higher-order spatial correlations and they have the same structural features that create salience in natural images. In Chapter 2, we evaluated human texture discrimination using 10 novel isotrigon textures (VnL2) and 17 standard V3L2 isotrigon textures. Factor analysis revealed that as few as 3 mechanisms may govern the detection of fourth- and higher-order image structure. The Maddess group has previously published evidence that the number of independent mechanisms is less than 10 and perhaps as small as 3-4. The computation of higher-order correlations by the brain is neuro-physiologically plausible via nonlinear combinations of recursive and/or rectifying processes. In Chapter 3, we utilised the crowdsourcing platform “mTurk” to implement a large texture discrimination study. Under laboratory conditions, we showed that the testing modality was robust across a range of browsers, resolutions, contrasts and screen sizes. Texture discrimination data was gathered from 121 naïve subjects and compared to 2 independent laboratory data sets. Factor analysis indicated the presence of 3-4 factors, consistent with previous studies. Based on Pearson's correlation and coefficients of repeatability, mTurk is capable of producing data of comparable quality to laboratory studies. This is significant as mTurk has not previously been systematically evaluated for visual psychometric research. In Chapter 4, we employed a set of statistically controlled ternary textures. The textures were constrained (spatial correlations from 1st to 4th order) and their salience could be independently controlled by the addition of noise. To the ideal observer, all textures defined by a given amount of noise are equally detectable. However, humans are not ideal observers; their visual perceptual resources are restricted. Because of the number of textures available, we used mTurk to gather performance functions from 928 subjects for a subset of the texture space. Perceptual salience varied for each image statistic, with rank order: gamma > beta_hv > beta_diag > alpha > theta. This supports the order previously published for the related binary stochastic textures. The two least salient directions were consistently white:black and grey-bias (for gammas and betas), and black:grey and grey:white (for thetas and alphas). Such differences reflect the sensitivities and limitations of neural processing and are a manifestation of efficient coding. We hypothesised that the grey token conferred non-salience. Indeed, for gammas and betas, the grey-bias was consistently the second least salient. However, this did not hold for thetas or alphas. Counter-intuitively, the order of texture presentation did not significantly affect discrimination performance. An analysis of 31 repeat Workers found evidence of learning for beta textures, whereas performance for other textures was already maximal. This thesis concludes by considering future research

A Study of Ethics in Crowd Work-Based Research

Crowd work as a form of a social-technical system has become a popular setting for conducting and distributing academic research. Crowd work platforms such as Amazon Mechanical Turk (MTurk) are widely used by academic researchers. Recent scholarship has highlighted the importance of ethical issues because they could affect the long-term development and application of crowd work in various fields such as the gig economy. However, little study or deliberation has been conducted on the ethical issues associated with academic research in this context. Current sources for ethical research practice, such as the Belmont Report, have not been examined thoroughly on how they should be applied to tackle the ethical issues in crowd work-based research such as those in data collection and usage. Hence, how crowd work-based research should be conducted to make it respectful, beneficent, and just is still an open question. This dissertation research has pursued this open question by interviewing 15 academic researchers and 17 IRB directors and analysts in terms of their perceptions and reflections on ethics in research on MTurk; meanwhile, it has analyzed 15 research guidelines and consent templates for research on MTurk and 14 published papers from the interviewed scholars. Based on analyzing these different sources of data, this dissertation research has identified three dimensions of ethics in crowd work-based research, including ethical issues in payment, data, and human subjects. This dissertation research also uncovered the “original sin” of these ethical issues and discussed its impact in academia, as well as the limitations of the Belmont Report and AoIR Ethical Guidelines 3.0 for Internet Research. The findings and implications of this research can help researchers and IRBs be more conscious about ethics in crowd work-based research and also inspire academic associations such as AoIR to develop ethical guidelines that can address these ethical issues

Velum: A 3D Puzzle Game and Facial Analysis Study

Velum is a first-person 3D puzzle/exploration game set in a timeless version of the Boston Public Garden. The project’s narrative framework and aesthetics are based on one of the Garden’s most prominent features, the Ether Monument, which commemorates the 1846 discovery of diethyl ether’s effectiveness as a medical anesthetic. A sequence of nine abstract challenges is rewarded by a progressive revelation of the player’s mysterious identity and purpose. The puzzle design was informed by the use of crowdsourced playtesting involving 300+ volunteers, combining standard data telemetry with AI-based facial image analysis capable of mapping player emotions to gameplay events

Velum: A 3D Puzzle Game and Facial Analysis Study

Velum is a first-person 3D puzzle/exploration game set in a timeless version of the Boston Public Garden. The project’s narrative framework and aesthetics are based on one of the Garden’s most prominent features, the Ether Monument, which commemorates the 1846 discovery of diethyl ether’s effectiveness as a medical anesthetic. A sequence of nine abstract challenges is rewarded by a progressive revelation of the player’s mysterious identity and purpose. The puzzle design was informed by the use of crowdsourced playtesting involving 300+ volunteers, combining standard data telemetry with AI-based facial image analysis capable of mapping player emotions to gameplay events

SOCIAL CLASS AND EMPLOYABILITY: EQUALIZING PERCEIVED COMPETENCE AND WARMTH TO CONTROL BIASED DECISION-MAKING DURING RESUMÉ SCREENING

Implicit bias during the resumé screening process can adversely impact the ability of an organization to achieve a competitive advantage through human capital (Coff & Kryscynski, 2011). The purpose of this study was to determine if teaching resumé screeners how to control biased decision-making during resumé screening results in equal employability ratings for upper-middle and lower-middle-class applicants. The study used a quantitative, causal, quasi-experimental, single-group pretest-post-test design. The target population was people in the United States who screen resumés as part of their current job duties (Thomas, 2018). The researcher used Amazon Mechanical Turk (MTurk) to recruit participants. Participants received a job description for a management training program and two resumés, one representing an upper-middle-class job applicant and one representing a lower-middle-class applicant (Thomas, 2018). Participants rated each resumé on perceptions of warmth, competence, and employability using the warmth and competence scales (Fiske, 2018) and an Employment Assessment scale (Cole et al., 2009). Participants viewed four short training videos that included two tactics to reduce biased behavior (Carter et al., 2020; Devine et al., 2012). After treatment, the researcher repeated the pretest procedure, and participants received two new resumés to rate. At the pretest, employability ratings were not significantly different between upper-middle-class and lower-middle-class applicants. At the post-test, participants rated the lower-middle-class applicant higher for employability. Perceived competence mediated the effect of social class on employability at the pretest and again at the post-test. Perceived warmth mediated the effect of social class on employability only at the post-test