Understanding information diversity in the era of repurposable crowdsourced data

Organizations successfully leverage information technology for the acquisition of knowledge for decision-making through information crowdsourcing, which is gathering information from a group of people about a phenomenon of interest to the crowdsourcer. Information crowdsourcing has been used to drive business insight and scientific research, providing crowdsourcers access to information outside their traditional reach. Crowdsourcers seek high-quality data for their information crowdsourcing projects and require contributors who can provide data that meet predetermined requirements. Crowdsourcers recruit contributors with high levels of relevant knowledge or train contributors to ensure the quality of data they collect. However, when crowdsourced data needs to fit more than a single usage scenario because the requirements of the project changed or the data needs to be repurposed for tasks other than the one(s) for which it was initially collected, the ability of contributors to provide diverse data that can meet multiple requirements is also desirable. In this thesis, I investigate how the domain knowledge a contributor possesses affects the diversity and quality of data they report. Using an experiment in which 84 students randomly assigned to three knowledge conditions reported information about artificial stimuli, I found that explicitly trained contributors provided less diverse data than either implicitly trained or untrained contributors. In addition, I looked at the longitudinal effect of knowledge on the diversity of data reported by contributors. Using review data from Amazon.com and organism sighting data from NLNature.com (a citizen science data crowdsourcing platform), I studied the impact of knowledge on the diversity and quality of crowdsourced data. The results show that experience reduced the diversity and usefulness of contributed data. The study provides insights for crowdsourcers in industry and academia on how to manage and utilize their crowds effectively to collect high-quality reusable data

On the adoption of crowdsourcing for theory testing

This paper examines the possibilities of using the crowdsourcing strategy for theory testing. We first analyse the relationships between theory building and theory testing activities. Then, based on a systematic review of 248 papers published in MISQ, we characterise the intents and pat-tern systems of activities that have been used for theory testing. Finally, we ascertain which ac-tivities can be crowdsourced or not and pinpoint a set of pathways supporting partial and total crowdsourcing. The obtained results show that a large number of activities related to data gath-ering can be crowdsourced, and that a number of intents have viable pathways supporting par-tial crowdsourcing

Crowdsourcing Design: A Synthesis of Literatures

Crowdsourcing is a phenomenon emerging in various sectors and industries that provides an opportunity for governments to collaborate with the public to generate information, deliver public services, or facilitate policy innovation. This review paper synthesizes prior research and practices on crowdsourcing from a variety of disciplines and focuses on the purpose, crowd, motivation, process design and outcomes. A process map for governments to design crowdsourcing is generated and three key actions are highlighted, namely incentive design, communication, and information aggregation

A Multi-Dimensional Approach for Framing Crowdsourcing Archetypes

All different kinds of organizations – business, public, and non-governmental alike – are becoming aware of a soaring complexity in problem solving, decision making and idea development. In a multitude of circumstances, multidisciplinary teams, high-caliber skilled resources and world-class computer suites do not suffice to cope with such a complexity: in fact, a further need concerns the sharing and ‘externalization’ of tacit knowledge already existing in the society. In this direction, participatory tendencies flourishing in the interconnected society in which we live today lead ‘collective intelligence’ to emerge as key ingredient of distributed problem solving systems going well beyond the traditional boundaries of organizations. Resulting outputs can remarkably enrich decision processes and creative processes carried out by indoor experts, allowing organizations to reap benefits in terms of opportunity, time and cost. Taking stock of the mare magnum of promising opportunities to be tapped, of the inherent diversity lying among them, and of the enormous success of some initiative launched hitherto, the thesis aspires to provide a sound basis for the clear comprehension and systematic exploitation of crowdsourcing. After a thorough literature review, the thesis explores new ways for formalizing crowdsourcing models with the aim of distilling a brand-new multi-dimensional framework to categorize various crowdsourcing archetypes. To say it in a nutshell, the proposed framework combines two dimensions (i.e., motivations to participate and organization of external solvers) in order to portray six archetypes. Among the numerous significant elements of novelty brought by this framework, the prominent one is the ‘holistic’ approach that combines both profit and non-profit, trying to put private and public sectors under a common roof in order to examine in a whole corpus the multi-faceted mechanisms for mobilizing and harnessing competence and expertise which are distributed among the crowd. Looking at how the crowd may be turned into value to be internalized by organizations, the thesis examines crowdsourcing practices in the public as well in the private sector. Regarding the former, the investigation leverages the experience into the PADGETS project through action research – drawing on theoretical studies as well as on intensive fieldwork activities – to systematize how crowdsourcing can be fruitfully incorporated into the policy lifecycle. Concerning the private realm, a cohort of real cases in the limelight is examined – having recourse to case study methodology – to formalize different ways through which crowdsourcing becomes a business model game-changer. Finally, the two perspectives (i.e., public and private) are coalesced into an integrated view acting as a backdrop for proposing next-generation governance model massively hinged on crowdsourcing. In fact, drawing on archetypes schematized, the thesis depicts a potential paradigm that government may embrace in the coming future to tap the potential of collective intelligence, thus maximizing the utilization of a resource that today seems certainly underexploited

Combining crowd worker, algorithm, and expert efforts to find boundaries of objects in images

While traditional approaches to image analysis have typically relied upon either manual annotation by experts or purely-algorithmic approaches, the rise of crowdsourcing now provides a new source of human labor to create training data or perform computations at run-time. Given this richer design space, how should we utilize algorithms, crowds, and experts to better annotate images? To answer this question for the important task of finding the boundaries of objects or regions in images, I focus on image segmentation, an important precursor to solving a variety of fundamental image analysis problems, including recognition, classification, tracking, registration, retrieval, and 3D visualization. The first part of the work includes a detailed analysis of the relative strengths and weaknesses of three different approaches to demarcate object boundaries in images: by experts, by crowdsourced laymen, and by automated computer vision algorithms. The second part of the work describes three hybrid system designs that integrate computer vision algorithms and crowdsourced laymen to demarcate boundaries in images. Experiments revealed that hybrid system designs yielded more accurate results than relying on algorithms or crowd workers alone and could yield segmentations that are indistinguishable from those created by biomedical experts. To encourage community-wide effort to continue working on developing methods and systems for image-based studies which can have real and measurable impact that benefit society at large, datasets and code are publicly-shared (http://www.cs.bu.edu/~betke/BiomedicalImageSegmentation/)

Investigating Avatar Customization as a Motivational Design Strategy for Improving Engagement with Technology-Enabled Services for Health

Technology-enabled services for physical and mental health are a promising approach to improve healthcare globally. Unfortunately, the largest barrier for effective technology-based treatment is participants' gradually fading engagement with effective novel training applications, such as exercise apps or online mental health training programs. Engaging users through design presents an elegant solution to the problem; however, research on technology-enabled services is primarily focused on the efficacy of novel interventions and not on improving adherence through engaging interaction design. As a result, motivational design strategies to improve engagement---both in the moment of use and over time---are underutilized. Drawing from game-design, I investigate avatar customization as a game-based motivational design strategy in four studies. In Study 1, I examine the effect of avatar customization on experience and behaviour in an infinite runner game. In Study 2, I induce different levels of motivation to research the effects of financial rewards on self-reported motivation and performance in a gamified training task over 11 days. In Study 3, I apply avatar customization to investigate the effects of attrition in an intervention context using a breathing exercise over three weeks. In Study 4, I investigate the immediate effects of avatar customization on the efficacy of an anxiety reducing attentional retraining task. My results show that avatar customization increases motivation over time and in the moment of use, suggesting that avatar customization is a viable strategy to address the engagement barrier that thwarts the efficacy of technology-enabled services for health

Combining crowd worker, algorithm, and expert efforts to find boundaries of objects in images

While traditional approaches to image analysis have typically relied upon either manual annotation by experts or purely-algorithmic approaches, the rise of crowdsourcing now provides a new source of human labor to create training data or perform computations at run-time. Given this richer design space, how should we utilize algorithms, crowds, and experts to better annotate images? To answer this question for the important task of finding the boundaries of objects or regions in images, I focus on image segmentation, an important precursor to solving a variety of fundamental image analysis problems, including recognition, classification, tracking, registration, retrieval, and 3D visualization. The first part of the work includes a detailed analysis of the relative strengths and weaknesses of three different approaches to demarcate object boundaries in images: by experts, by crowdsourced laymen, and by automated computer vision algorithms. The second part of the work describes three hybrid system designs that integrate computer vision algorithms and crowdsourced laymen to demarcate boundaries in images. Experiments revealed that hybrid system designs yielded more accurate results than relying on algorithms or crowd workers alone and could yield segmentations that are indistinguishable from those created by biomedical experts. To encourage community-wide effort to continue working on developing methods and systems for image-based studies which can have real and measurable impact that benefit society at large, datasets and code are publicly-shared (http://www.cs.bu.edu/~betke/BiomedicalImageSegmentation/)