Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actions

Crowdsourcing enables one to leverage on the intelligence and wisdom of potentially large groups of individuals toward solving problems. Common problems approached with crowdsourcing are labeling images, translating or transcribing text, providing opinions or ideas, and similar - all tasks that computers are not good at or where they may even fail altogether. The introduction of humans into computations and/or everyday work, however, also poses critical, novel challenges in terms of quality control, as the crowd is typically composed of people with unknown and very diverse abilities, skills, interests, personal objectives and technological resources. This survey studies quality in the context of crowdsourcing along several dimensions, so as to define and characterize it and to understand the current state of the art. Specifically, this survey derives a quality model for crowdsourcing tasks, identifies the methods and techniques that can be used to assess the attributes of the model, and the actions and strategies that help prevent and mitigate quality problems. An analysis of how these features are supported by the state of the art further identifies open issues and informs an outlook on hot future research directions.Comment: 40 pages main paper, 5 pages appendi

Designing for Collective Intelligence and Community Resilience on Social Networks

The popularity and ubiquity of social networks has enabled a new form of decentralised online collaboration: groups of users gathering around a central theme and working together to solve problems, complete tasks and develop social connections. Groups that display such ‘organic collaboration’ have been shown to solve tasks quicker and more accurately than other methods of crowdsourcing. They can also enable community action and resilience in response to different events, from casual requests to emergency response and crisis management. However, engaging such groups through formal agencies risks disconnect and disengagement by destabilising motivational structures. This paper explores case studies of this henomenon, reviews models of motivation that can help design systems to harness these groups and proposes a framework for lightweight engagement using existing platforms and social networks

Sustaining public agency in caring for heritage: critical perspectives on participation through co-design

This thesis explores how heritage organisations in the UK are attempting to build capacity and sustainability in community groups involved in caring for heritage places during austerity. It is based on a broad interdisciplinary reading of critical perspectives on public participation. From this vantage point, I argue that the forms of participation facilitated by participatory initiatives in the sector are constrained by perceptions of public deficits and legitimate heritage expertise, which in turn are bound up in established definitions of heritage and its cultural significance. As a result, participatory initiatives reproduce the characteristics of network governance and incumbent democratisation, whereby community groups who share professional values are asked to augment professional capacity, as opposed to more critical forms of democratisation that foreground public agency. By critically engaging with my three case studies, Archaeology Scotland’s Adopt-a-Monument scheme, Bristol City Council’s Know Your Place interface and associated projects and my own co-design project with three community groups in Yorkshire, I demonstrate how public agency is limited in practice in each case, despite individuals’ critical intentions. In response, I argue that increasing and sustaining public agency in caring for heritage requires carefully designing participatory projects in ways that foreground participants’ skills and interests. My analysis demonstrates that in order to realise such interventions, they must be based in reconceptualised definitions of heritage and more nuanced understandings of participation deficits and legitimate heritage expertise. In doing so, my thesis contributes to the growing body of scholarship that argues increasing public participation is not a critical intervention in and of itself, but a means by which control can be both retained and relinquished

TalkFutures: Supporting Qualitative Practices in Distributed Community Engagements

Community engagements are qualitative processes that make use of participants local knowledge for democratic decision-making, but often exclude participants from data analysis and dissemination. This can mean that they are left feeling that their voice is not properly represented in the final output. This paper presents a digital community engagement process, TalkFutures, that actively involves participants in the production, distributed analysis and summarization of qualitative data. The design of TalkFutures was explored through a five-week deployment with the International Federation of Red Cross and Red Crescent Societies (IFRC) as part of a consultation designed to inform future strategy. Our analysis of deployment metrics and post-deployment interviews outline how TalkFutures: (i) increased modes of participation across the qualitative workflow; (ii) reduced barriers to participation; and (iii) improved representation in the engagement processes

Making digital history: The impact of digitality on public participation and scholarly practices in historical research

This thesis investigates tow key questions: firstly, how do two broad groups - academic, family and local historians, and the public - evaluate, use, and contribute to digital history resources? And consequently, what impact have digital technologies had on public participation and scholarly practices in historical research? Analysing the impact of design on participant experiences and the reception of digital historiography by demonstrating the value of methods drawn from human-computer interaction, including heuristic evaluation, trace ethnography and semi-structured interviews. This thesis also investigates the relationship between heritage crowdsourcing projects (which ask the public to help with meaningful, inherently rewarding tasks that contribute to a shared, significant goal or research interest related to cultural heritage collections or knowledge) and the development of historical skills and interests. It situates crowdsourcing and citizen history within the broader field of participatory digital history and then focuses on the impact of digitality on the research practices of faculty and community historians. Chapter 1 provides an overview of over 400 digital history projects aimed at engaging the public or collecting, creating or enhancing records about historical materials for scholarly and general audiences. Chapter 2 discusses design factors that may influence the success of crowdsourcing projects. Following this, Chapter 3 explores the ways in which some crowdsourcing projects encourage deeper engagement with history or science, and the role of communities of practice in citizen history. Chapter 4 shifts our focus from public participation to scholarly practices in historical research, presenting the results of interviews conducted with 29 faculty and community historians. Finally, the Conclusion draws together the threads that link public participation and scholarly practices, teasing out the ways in which the practices of discovering, gathering, creating and sharing historical materials and knowledge have been affected by digital methods, tools and resources

A Task-Interdependency Model of Complex Collaboration Towards Human-Centered Crowd Work

Models of crowdsourcing and human computation often assume that individuals independently carry out small, modular tasks. However, while these models have successfully shown how crowds can accomplish significant objectives, they can inadvertently advance a less than human view of crowd workers and fail to capture the unique human capacity for complex collaborative work. We present a model centered on interdependencies -- a phenomenon well understood to be at the core of collaboration -- that allows one to formally reason about diverse challenges to complex collaboration. Our model represents tasks as an interdependent collection of subtasks, formalized as a task graph. We use it to explain challenges to scaling complex collaborative work, underscore the importance of expert workers, reveal critical factors for learning on the job, and explore the relationship between coordination intensity and occupational wages. Using data from O*NET and the Bureau of Labor Statistics, we introduce an index of occupational coordination intensity to validate our theoretical predictions. We present preliminary evidence that occupations with greater coordination intensity are less exposed to displacement by AI, and discuss opportunities for models that emphasize the collaborative capacities of human workers, bridge models of crowd work and traditional work, and promote AI in roles augmenting human collaboration

Capturing volunteered information for inclusive service design: potential benefits and challenges

Inclusive Design focuses on understanding the broad spectrum of peoples’ needs and abilities, with a view to developing more successful products and services. However, peoples’ experiences with products and services are dynamic and multi-layered, presenting a unique set of challenges for Inclusive Designers. This paper presents the concept of Volunteered Geographic Information (VGI) as an integral part of future inclusive services. By utilising crowd-sourced data, services can become more efficient, intuitive and relevant for a wider population than previously possible. The potential benefits and challenges are presented and explored through a series of qualitative case studies. These focus on the differences in data generated by disabled and older people, and the uniqueness of the information gained. This type of information has the potential to provide a better match between user needs and service delivery, and enable the successful longer-term evolution of services

Public Archaeology in a Digital Age

This thesis examines the impact of the democratic promises of Internet communication technologies, social, and participatory media on the practice of public archaeology. It is focused on work within archaeological organisations in the UK in commercial archaeology, higher education, local authority planning departments and community settings, as well the voluntary planning departments and community settings, as well the voluntary archaeology sector archaeology sector . This work has taken an innovative approach to the subject matter through its use of a Grounded Theory method for data collection and analysis, and the use of a combination of online surveys, case studies and email questionnaires in order to address the following issues: the provision of authoritative archaeological information online; barriers to participation; policy and organisational approaches to evaluating success and archiving; community formation and activism, and the impact of digital inequalities and literacies. This thesis is the first overarching study into the use of participatory media in archaeology. It is an important exploration of where and how the profession is creating and managing digital platforms, and the expanding opportunities for networking and sharing information within the discipline, against a backdrop of rapid advancement in the use of Internet technologies within society. This work has made significant contributions to debates on the practice and impact of public archaeology. It has shown that archaeologists do not yet fully understand the complexities of Internet use and issues of digital literacy, the impact of audience demographics or disposition towards participation in online projects. It has shown that whilst recognition of democratic participation is not, on the whole, undertaken through a process of actively acknowledging responses to archaeological information, there remains potential for participatory media to support and accommodate these ideals. This work documents a period of great change within the practice of archaeology in the UK, and concludes with the observation that it is vital that the discipline undertake research into online audiences for archaeological information if we are to create sustainable digital public archaeologies