Spam elimination and bias correction : ensuring label quality in crowdsourced tasks.

Crowdsourcing is proposed as a powerful mechanism for accomplishing large scale tasks via anonymous workers online. It has been demonstrated as an effective and important approach for collecting labeled data in application domains which require human intelligence, such as image labeling, video annotation, natural language processing, etc. Despite the promises, one big challenge still exists in crowdsourcing systems: the difficulty of controlling the quality of crowds. The workers usually have diverse education levels, personal preferences, and motivations, leading to unknown work performance while completing a crowdsourced task. Among them, some are reliable, and some might provide noisy feedback. It is intrinsic to apply worker filtering approach to crowdsourcing applications, which recognizes and tackles noisy workers, in order to obtain high-quality labels. The presented work in this dissertation provides discussions in this area of research, and proposes efficient probabilistic based worker filtering models to distinguish varied types of poor quality workers. Most of the existing work in literature in the field of worker filtering either only concentrates on binary labeling tasks, or fails to separate the low quality workers whose label errors can be corrected from the other spam workers (with label errors which cannot be corrected). As such, we first propose a Spam Removing and De-biasing Framework (SRDF), to deal with the worker filtering procedure in labeling tasks with numerical label scales. The developed framework can detect spam workers and biased workers separately. The biased workers are defined as those who show tendencies of providing higher (or lower) labels than truths, and their errors are able to be corrected. To tackle the biasing problem, an iterative bias detection approach is introduced to recognize the biased workers. The spam filtering algorithm proposes to eliminate three types of spam workers, including random spammers who provide random labels, uniform spammers who give same labels for most of the items, and sloppy workers who offer low accuracy labels. Integrating the spam filtering and bias detection approaches into aggregating algorithms, which infer truths from labels obtained from crowds, can lead to high quality consensus results. The common characteristic of random spammers and uniform spammers is that they provide useless feedback without making efforts for a labeling task. Thus, it is not necessary to distinguish them separately. In addition, the removal of sloppy workers has great impact on the detection of biased workers, with the SRDF framework. To combat these problems, a different way of worker classification is presented in this dissertation. In particular, the biased workers are classified as a subcategory of sloppy workers. Finally, an ITerative Self Correcting - Truth Discovery (ITSC-TD) framework is then proposed, which can reliably recognize biased workers in ordinal labeling tasks, based on a probabilistic based bias detection model. ITSC-TD estimates true labels through applying an optimization based truth discovery method, which minimizes overall label errors by assigning different weights to workers. The typical tasks posted on popular crowdsourcing platforms, such as MTurk, are simple tasks, which are low in complexity, independent, and require little time to complete. Complex tasks, however, in many cases require the crowd workers to possess specialized skills in task domains. As a result, this type of task is more inclined to have the problem of poor quality of feedback from crowds, compared to simple tasks. As such, we propose a multiple views approach, for the purpose of obtaining high quality consensus labels in complex labeling tasks. In this approach, each view is defined as a labeling critique or rubric, which aims to guide the workers to become aware of the desirable work characteristics or goals. Combining the view labels results in the overall estimated labels for each item. The multiple views approach is developed under the hypothesis that workers\u27 performance might differ from one view to another. Varied weights are then assigned to different views for each worker. Additionally, the ITSC-TD framework is integrated into the multiple views model to achieve high quality estimated truths for each view. Next, we propose a Semi-supervised Worker Filtering (SWF) model to eliminate spam workers, who assign random labels for each item. The SWF approach conducts worker filtering with a limited set of gold truths available as priori. Each worker is associated with a spammer score, which is estimated via the developed semi-supervised model, and low quality workers are efficiently detected by comparing the spammer score with a predefined threshold value. The efficiency of all the developed frameworks and models are demonstrated on simulated and real-world data sets. By comparing the proposed frameworks to a set of state-of-art methodologies, such as expectation maximization based aggregating algorithm, GLAD and optimization based truth discovery approach, in the domain of crowdsourcing, up to 28.0% improvement can be obtained for the accuracy of true label estimation

The Dawn of Crowdfarms

Some small companies are making crowdwork part of their formal business via teams that can complete multifaceted, complex tasks requiring specialized expertise

An Examination of the Work Practices of Crowdfarms

Crowdsourcing is a new value creation business model. Annual revenue of the Chinese market alone is hundreds of millions of dollars, yet few studies have focused on the practices of the Chinese crowdsourcing workforce, and those that do mainly focus on solo crowdworkers. We have extended our study of solo crowdworker practices to include crowdfarms, a relatively new entry to the gig economy: small companies that carry out crowdwork as a key part of their business. We report here on interviews of people who work in 53 crowdfarms. We describe how crowdfarms procure jobs, carry out macrotasks and microtasks, manage their reputation, and employ different management practices to motivate crowdworkers and customers

From Business Model to Business Modelling: Modularity and Manipulation

The concept of modularity has gained considerable traction in technology studies as a way to conceive, describe and innovate complex systems, such as product design or organizational structures. In the recent literature, technological modularity has often been intertwined with business model innovation, and scholarship has started investigating how modularity in technology affects changes in business models, both at the cognitive and activity system levels. Yet we still lack a theoretical definition of what modularity is in the business model domain. Business model innovation also encompasses different possibilities of modeling businesses, which are not clearly understood nor classified. We ask when, how and if modularity theory can be extended to business models in order to enable effective and efficient modeling. We distinguish theoretically between modularity for technology and for business models, and investigate the key processes of modularization and manipulation. We introduce the basic operations of business modeling via modular operators adapted from the technological modularity domain, using iconic examples to develop an analogical reasoning between modularity in technology and in business models. Finally, we discuss opportunities for using modularity theory to foster the understanding of business models and modeling, and develop a challenging research agenda for future investigations

Pika: Empowering Non-Programmers to Author Executable Governance Policies in Online Communities

Internet users have formed a wide array of online communities with nuanced and diverse community goals and norms. However, most online platforms only offer a limited set of governance models in their software infrastructure and leave little room for customization. Consequently, technical proficiency becomes a prerequisite for online communities to build governance policies in code, excluding non-programmers from participation in designing community governance. In this paper, we present Pika, a system that empowers non-programmers to author a wide range of executable governance policies. At its core, Pika incorporates a declarative language that decomposes governance policies into modular components, thereby facilitating expressive policy authoring through a user-friendly, form-based web interface. Our user studies with 17 participants show that Pika can empower non-programmers to author governance policies approximately 2.5 times faster than programmers who author in code. We also provide insights about Pika's expressivity in supporting diverse policies that online communities want.Comment: Under revie

Governance of Inter-Organizational Collaborations When Engaged in Open Innovation

L'ús del coneixement extern, a través de projectes oberts i de col·laboració amb els socis externs, permet a les empreses resoldre amb eficàcia i eficiència els seus problemes d'innovació, generant així un major acompliment de la innovació. No obstant, molts projectes d'innovació oberta i de col·laboració no han pogut completar els seus objectius, tal com s'havia planejat inicialment. Els acadèmics han tractat d'examinar aquest problema mitjançant l'estudi del mecanisme de govern del procés de col·laboració en les fases de formació i execució dels projectes. Si bé aquests estudis han aportat coneixements importants, encara es sap poc sobre la naturalesa de la dinàmica de col·laboració i els atributs dels projectes que afecten els mecanismes de govern. En resposta, aquesta tesi pretén establir una visió àmplia i clarificadora del govern de la innovació oberta i de col·laboració a través de l'abordatge de la següent pregunta general: com les empreses governen el procés de col·laboració amb socis externs per a augmentar la probabilitat que els seus projectes d'innovació oberta i de col·laboració es completin amb èxit? Tres preguntes específiques d'investigació es determinen per respondre a la pregunta general: 1) ¿com gestionen les empreses la dinàmica del procés de col·laboració amb fonts externes per completar amb èxit els seus projectes d'innovació oberta i de col·laboració? 2) ¿l'ús d'un procés formalitzat conjunt de tecnologia i desenvolupament ajuda a augmentar la probabilitat que un projecte d'innovació oberta amb fonts externes es completi amb èxit? 3) ¿quins modes d'innovació oberta trien els directius per a projectes que es caracteritzen per diferents nivells de complexitat i 'ocultació' del coneixement? Responem a aquestes preguntes combinant un anàlisi sistemàtic de casos creuats dels casos qualitatius de projectes oberts i de col·laboració i un estudi d'enquesta. Els resultats d'aquest estudi demostren que les empreses associades han de regular la tensió entre compartir i protegir el coneixement en els processos de col·laboració per a completar amb èxit projectes conjunts. D'altra banda, presento una forma alternativa de formalització en el procés de col·laboració, a més del control de la propietat intel·lectual (IP), per a regular la tensió entre compartir i protegir el coneixement. Finalment, els resultats indiquen que els atributs del projecte, específicament la complexitat i l'ocultació del coneixement necessari, afecten la selecció dels mecanismes de govern en projectes oberts i de col·laboració.El uso del conocimiento externo, a través de proyectos abiertos y de colaboración con los socios externos, permite a las empresas resolver con eficacia y eficiencia sus problemas de innovación, generando así un mayor desempeño de la innovación. Sin embargo, muchos proyectos de innovación abierta y de colaboración no han podido completar sus objetivos, tal y como se había planeado inicialmente. Los académicos han tratado de examinar este problema mediante el estudio del mecanismo de gobierno del proceso de colaboración en las fases de formación y ejecución de los proyectos. Si bien estos estudios han aportado conocimientos importantes, aún se sabe poco acerca de la naturaleza de la dinámica de colaboración y los atributos de los proyectos que afectan a los mecanismos de gobierno. En respuesta, esta tesis pretende establecer una visión amplia y clarificadora del gobierno de la innovación abierta y de colaboración a través del abordaje de la siguiente pregunta general: ¿cómo las empresas gobiernan el proceso de colaboración con socios externos para aumentar la probabilidad de que sus proyectos de innovación abierta y de colaboración se completen con éxito? Tres preguntas específicas de investigación se determinan para responder a la pregunta general: 1) ¿cómo gestionan las empresas la dinámica del proceso de colaboración con fuentes externas para completar con éxito sus proyectos de innovación abierta y de colaboración? 2) ¿el uso de un proceso formalizado conjunto de tecnología y desarrollo ayuda a aumentar la probabilidad de que un proyecto de innovación abierta con fuentes externas se complete con éxito? 3) ¿qué modos de innovación abierta escogen los directivos para proyectos que se caracterizan por diferentes niveles de complejidad y 'ocultamiento' del conocimiento? Respondemos a estas preguntas combinando un análisis sistemático de casos cruzados de los casos cualitativos de proyectos abiertos y de colaboración y un estudio de encuesta. Los resultados de este estudio demuestran que las empresas asociadas tienen que regular la tensión entre compartir y proteger el conocimiento en los procesos de colaboración para completar con éxito proyectos conjuntos. Por otra parte, presento una forma alternativa de formalización en el proceso de colaboración, además del control de la propiedad intelectual (IP), para regular la tensión entre compartir y proteger el conocimiento. Por último, los resultados indican que los atributos del proyecto, específicamente la complejidad y el ocultamiento del conocimiento necesario, afectan a la selección de los mecanismos de gobierno en proyectos abiertos y de colaboración.The use of external knowledge, through open and collaborative projects with external partners, enables firms to effectively and efficiently solve their innovation problems, thereby generating greater innovation performance. Yet many open and collaborative innovation projects have failed to complete their objectives as initially planned. Scholars have tried to examine this problem by studying the governance mechanism of collaboration process in both formation and execution phases of projects. While these studies have provided important insights, still little is understood about the nature of collaboration dynamics and the attributes of projects affecting governance mechanisms. In response, this dissertation seeks to establish a comprehensive and clarifying view of open and collaborative innovation governance through addressing the following overall question: How do firms govern the collaboration process with external partners to increase the likelihood that their open and collaborative innovation projects are successfully completed? Three specific research questions are framed to answer the overall question: 1) How do firms manage the dynamics of collaboration process with external sources to successfully complete their open and collaborative innovation projects? 2) Does the use of a formalized joint technology-development process help to increase the likelihood that an open innovation project with external sources is successfully completed? 3) Which open innovation modes do managers choose for projects characterized by different levels of complexity and ‘hiddenness’ of knowledge? We approach these questions with combining a cross-case systematic analysis of qualitative cases on open and collaborative projects and a survey study. The results of this study demonstrate that partnering firms need to regulate the knowledge sharing-protecting tension in collaboration processes to successfully complete joint projects. Moreover, I introduce an alternative form of formalization into the collaboration process, in addition to formal intellectual property (IP) control, to regulate the knowledge sharing-protecting tension. Finally, the results indicate that project attributes, specifically complexity and hiddenness of required knowledge, affects the selection of governance mechanisms in open and collaborative projects

Crowdsourcing Controls: A Review and Research Agenda for Crowdsourcing Controls Used for Macro-tasks

Crowdsourcing—the employment of ad hoc online labor to perform various tasks—has become a popular outsourcing vehicle. Our current approach to crowdsourcing—focusing on micro-tasks—fails to leverage the potential of crowds to tackle more complex problems. To leverage crowds to tackle more complex macro tasks requires a better comprehension of crowdsourcing controls. Crowdsourcing controls are mechanisms used to align crowd workers’ actions with predefined standards to achieve a set of goals and objectives. Unfortunately, we know very little about the topic of crowdsourcing controls directed at accomplishing complex macro tasks. To address issues associated with crowdsourcing controls formacro-tasks, this chapter has several objectives. First, it presents and discusses the literature on control theory. Second, this chapter presents a scoping literature review of crowdsourcing controls. Finally, the chapter identifies gaps and puts forth a research agenda to address these shortcomings. The research agenda focuses on understanding how to employ the controls needed to perform macro-tasking in crowds and the implications for crowdsourcing system designers.National Science Foundation grant CHS-1617820Peer Reviewedhttps://deepblue.lib.umich.edu/bitstream/2027.42/150493/1/Robert 2019 Preprint Chapter 3.pdfDescription of Robert 2019 Preprint Chapter 3.pdf : PrePrint Versio