All That Glitters is Gold -- An Attack Scheme on Gold Questions in Crowdsourcing

One of the most popular quality assurance mechanisms in paid micro-task crowdsourcing is based on gold questions: the use of a small set of tasks of which the requester knows the correct answer and, thus, is able to directly assess crowd work quality. In this paper, we show that such mechanism is prone to an attack carried out by a group of colluding crowd workers that is easy to implement and deploy: the inherent size limit of the gold set can be exploited by building an inferential system to detect which parts of the job are more likely to be gold questions. The described attack is robust to various forms of randomisation and programmatic generation of gold questions. We present the architecture of the proposed system, composed of a browser plug-in and an external server used to share information, and briefly introduce its potential evolution to a decentralised implementation. We implement and experimentally validate the gold detection system, using real-world data from a popular crowdsourcing platform. Finally, we discuss the economic and sociological implications of this kind of attack

Adversarial attacks on crowdsourcing quality control

Crowdsourcing is a popular methodology to collect manual labels at scale. Such labels are often used to train AI models and, thus, quality control is a key aspect in the process. One of the most popular quality assurance mechanisms in paid micro-task crowdsourcing is based on gold questions: the use of a small set of tasks of which the requester knows the correct answer and, thus, is able to directly assess crowd work quality. In this paper, we show that such mechanism is prone to an attack carried out by a group of colluding crowd workers that is easy to implement and deploy: the inherent size limit of the gold set can be exploited by building an inferential system to detect which parts of the job are more likely to be gold questions. The described attack is robust to various forms of randomisation and programmatic generation of gold questions. We present the architecture of the proposed system, composed of a browser plug-in and an external server used to share information, and briefly introduce its potential evolution to a decentralised implementation. We implement and experimentally validate the gold detection system, using real-world data from a popular crowdsourcing platform. Our experimental results show that crowd workers using the proposed system spend more time on signalled gold questions but do not neglect the others thus achieving an increased overall work quality. Finally, we discuss the economic and sociological implications of this kind of attack

Deep Bayesian Trust : A Dominant and Fair Incentive Mechanism for Crowd

An important class of game-theoretic incentive mechanisms for eliciting effort from a crowd are the peer based mechanisms, in which workers are paid by matching their answers with one another. The other classic mechanism is to have the workers solve some gold standard tasks and pay them according to their accuracy on gold tasks. This mechanism ensures stronger incentive compatibility than the peer based mechanisms but assigning gold tasks to all workers becomes inefficient at large scale. We propose a novel mechanism that assigns gold tasks to only a few workers and exploits transitivity to derive accuracy of the rest of the workers from their peers' accuracy. We show that the resulting mechanism ensures a dominant notion of incentive compatibility and fairness

The Dark Side of Recruitment in Crowdsourcing: Ethics and Transparency in Micro-Task Marketplaces

Micro-task crowdsourcing marketplaces like Figure Eight (F8) connect a large pool of workers to employers through a single online platform, by aggregating multiple crowdsourcing platforms (channels) under a unique system. This paper investigates the F8 channels’ demographic distribution and reward schemes by analysing more than 53k crowdsourcing tasks over four years, collecting survey data and scraping marketplace metadata. We reveal an heterogeneous per-channel demographic distribution, and an opaque channel commission scheme, that varies over time and is not communicated to the employer when launching a task: workers often will receive a smaller payment than expected by the employer. In addition, the impact of channel commission schemes on the relationship between requesters and crowdworkers is explored. These observations uncover important issues on ethics, reliability and transparency of crowdsourced experiment when using this kind of marketplaces, especially for academic research

The dark side of recruitment in crowdsourcing: ethics and transparency in micro-task marketplaces

Micro-task crowdsourcing marketplaces like Figure Eight (F8) connect a large pool of workers to employers through a single online platform, by aggregating multiple crowdsourcing platforms (channels) under a unique system. This paper investigates the F8 channels’ demographic distribution and reward schemes by analysing more than 53k crowdsourcing tasks over four years, collecting survey data and scraping marketplace metadata. We reveal an heterogeneous per-channel demographic distribution, and an opaque channel commission scheme, that varies over time and is not communicated to the employer when launching a task: workers often will receive a smaller payment than expected by the employer. In addition, the impact of channel commission schemes on the relationship between requesters and crowdworkers is explored. These observations uncover important issues on ethics, reliability and transparency of crowdsourced experiment when using this kind of marketplaces, especially for academic research

Understanding the Use of HIT Catchers and Crowd Knowledge Sharing: A Case Study of Crowdworkers on Amazon Mechanical Turk

Crowdsourcing platforms have become a vital component of the modern digital economy, offering a wide range of HIT (Human Intelligence Task) opportunities to workers worldwide. Meanwhile, crowdworkers' use of scripting tools and their communication with each other are continuously shaping the entire crowdsourcing ecosystem. This thesis explores the use of HIT catchers by crowdworkers and their sharing of skill-based knowledge that facilitates the popularity of such scripting tools. It is revealed that the use of HIT catchers affects the completion speed and HIT-worker diversity for the whole HIT group, while depriving job opportunities from others. This potentially undermines the stability of the platform under the current reputation system relying on numbers of approvals and approval rates. Subsequently, another study explored how work strategies under the use of HIT catchers, including HIT acceptance, backlog, and completion, affect HIT availability, completion time, and result quality. The study also found differences in work behaviours between workers using and not using HIT catchers. Finally, this thesis investigates the skill-based knowledge sharing behaviour of crowdworkers, which promotes the blooming of scripting tools including HIT catchers, to improve the fairness of work opportunities and mitigate its negative impact on HIT completion. Using PLS-SEM, we assess the factors influencing knowledge sharing in the domain of skills. The study reveals the significance of high performance expectation, low effort expectation, and the joy and satisfaction in motivating the crowd skill-based knowledge sharing. Overall, this study provides an in-depth exploration around these two types of collective behaviour, highlighting the important role of tool use and knowledge sharing in shaping the crowdsourcing ecosystem

A Matter of Material:Exploring the Value of the Museum of Design in Plastics

This thesis, A Matter of Material: Exploring the Value of the Museum of Design in Plastics (MoDiP), sets out to understand how a museum focusing on a single material family can contribute to the societal and museological comprehension of design in plastics. It looks at how museums communicate a group of materials that audiences believe they know and understand, yet that knowledge and understanding may not be the whole story. It explores why it might seem strange that a museum dedicated to plastics even exists, by looking at what museums are, what they have been traditionally, and what they can become. The contribution to knowledge that this research demonstrates is in the previously unwritten history and close study of MoDiP which is an, as yet, under researched resource. My role as Curator of MoDiP has provided an empirical knowledge and expertise that grounds this contribution in my professional practice. This has enabled an opening of a knowledge embedded in the role of a museum of a contested and devalued material, illuminating the problem of plastics in museums. The study inserts plastics, and the specific collection of them by MoDiP, into the relational museological theory to discover the value of the museum’s practice where complexity is added to the debate about plastics in the current climate. The particular interest of the triangular relationships between audiences, museums, and plastics is demonstrated using new diagrams, a tradition of museum studies especially used by Susan Pearce. The six original diagrams within the thesis are used to illustrate new ideas and concepts. This research uses the tools of case study as a methodology to make a close study of the functions and collections of MoDiP, and in contrast the Pinto Collection of wooden objects at Birmingham Museums Trust. These tools include interviewing employees, studying documents, and observing practices, and sit alongside the curatorial practices of collections and object research, audience sampling through surveys and social media, as well as visiting other museums and exhibitions and reflecting on such experiences. By using these methods, this work investigates the material qualities of plastics, alongside other materials, and looks at why the placement of some materials within the museum setting might be difficult to comprehend and how, by being the sole focus of the museum, materials can be more deeply explored

The tertiary education institution of the future towards 2030: scenarios for skills transformation

The research methodology used in this research was comprised of Inayatullah’s Six Pillars of Futures Studies, in which emphasis was placed on scenario planning and the creation of alternative scenarios for the tertiary education institutions in South Africa towards 2030. An environmental scan revealed the drivers of change in the education sector and in the world of work. Deepening of the future of education was done through Causal Layered Analysis (CLA) to facilitate the discerning of issues from various viewpoints in the creation and expansion of transformative stories so as to provide a window into possible futures for skills transformation. The four scenarios for the tertiary education institution of the future, namely “Stairway to Heaven”, “Highway to Hell”, “Bat out of Hell” and “Still Raining” were developed. These scenarios can be used as departure points by tertiary education providers to make strides towards the Global Sustainable Development Education 2030 targets and the attainment of South Africa’s Vision 2030 targets contained in the National Development Plan. Equally important, these scenarios make known what was previously unknown, exploring the possible and impossible, and encouraging new, innovative thinking for decision-makers. The “Stairway to Heaven” scenario supplies a future in which all stakeholders approve of and embrace the mandate of providing relevant skills and job readiness in a fast-changing world, and the benefits are maximised for all involved through co-creation. It is a scenario where industry, tertiary institutions and society have decided that the purpose of education should be lifelong learning for a viable, productive and sustainable world. The desired future of tertiary education is set against a backdrop of public and private sector collaboration, with the aim of turning the nation into an excellent hub for skills transformation. Furthermore, the scenario provides some insight on the vital measures required to embrace the innovation and the appropriate pedagogy. This research was motivated by the need to shine a light on the 21st century learner, rapidly obsoleting skills, no-collar worker, skills of the future, learning futures, and possible predictions about what new jobs may come into existence so that educationists can better prepare for the future. This research offers solutions on how institutions can prepare students for future jobs, especially considering the rapid changes in jobs and the unprecedented demise of certain jobs. The research closes a research gap through creating scenarios that offer various stakeholders in the tertiary education sector different insights and analysis into a number of interpretations of the potential paths that they can follow. The scenario application culminated in the formulation and creation of a “future vision of the tertiary education institution in South Africa towards 2030”, delivering a platform for skills transformation that will deliver adaptable workers, and sustainable and inclusive progress for all South Africans. To bring transformation into the present and design the future that embraces skills transformation, it is invaluable to interrogate the roles and choices that stakeholders of the educational sector make in determining the preferred future. The approach of this research makes it clear that, as the new world of work transpires, policymakers, students, labour, educational leaders, captains of industry and workers must proactively manage the workforce transitions. The focal issue is to discover the appropriate tools that will establish the confidence necessary to create the preferred future for skills transformation in tertiary institutions. This research has laid a platform for co-creation with various stakeholders in an effort to visualise a tertiary institution that contributes to skills development. The vision must accept that the South African jobs and skills historical profile is different from that of industrialised countries. Alternatively, the nation should respond to the double-barrelled challenge of participating in a high skills’ competitive environment on a global scale, as well as a local context that creates low-wage, blue-collar jobs to absorb the large numbers who are unemployed. The challenge is even greater for South Africa, because the economy – if highly service-oriented, with a big informal sector and a quality postgraduate education offering – is supported by a basic education system that is not producing enough critical thinkers who are equipped for university and work life. Thus, the system requires a double transformation to ensure student-centredness and meet the needs of a future worker

An Empty Promise? Digital Democracy in the Smart City

The digital transformation affects every part of our societies and everyday lives, including the processes and structures of our democracies. On the one hand, information and communication technologies have the potential to lower the threshold for political communication and participation. On the other hand, they can be used for large-scale data collection and surveillance, posing a risk to the public sphere. This thesis investigates the impact of digitization on the legitimacy of democracy. It first develops a novel framework based on the theories of participatory and deliberative democracy, drawing on recent work on deliberative systems. On this basis, digital democracy is examined as a system, consisting of different engagement spaces and actors within the smart city. The smart city is a particularly fruitful testbed for digital democracy as it is based on the promise of applying a high density of digital technologies to facilitate civic participation as well as better service delivery and governance. Through an in-depth case study of the smart city of Amsterdam, this thesis not only reveals the legitimacy dilemmas of digital democracy in the smart city, but also illustrates the limits of applying participatorydeliberative systems theory on a digital democracy ecosystem. The analysis demonstrates design conflicts between different online engagement platforms within the digital democracy system, as well as conflicting objectives among the actors behind them. The findings do not support the claim that digitization negatively impacts democracy’s legitimacy in the smart city of Amsterdam through marketization, large-scale data collection, and surveillance, as some authors warn. However, a significant positive impact of digitization on democratic legitimacy, through higher levels of inclusiveness, empowerment, or civic influence, is also not confirmed. The findings show that digital technologies’ promise of facilitating large-scale citizen participation and deliberation in the smart city does not live up to the normative ideal. The results from Amsterdam are exposed to smart city and digital democracy experts across the globe to test their generalizability, demonstrating that, despite its shortcomings, Amsterdam’s extensive digital democracy system is far advanced in international comparison. What may appear a contradiction in fact illustrates that we are still in the early stages of development, with potential to enhance the legitimacy of digital democracy, both in the smart city of Amsterdam and beyond

The datafied welfare state: a perspective from the UK

The crisis emerging from the COVID-19 pandemic has elevated the relevance of the welfare state as well as the role of platforms and data infrastructures across key areas of public and social life. Whilst the crisis shed light on the ways in which these might intersect, the turn to data-driven systems in public administration has been a prominent development in several countries for quite some time. In this chapter I focus on the UK as a pertinent example of key trends at the intersection of technological infrastructures and the welfare state. In particular, using developments in UK welfare sectors as a lens, I advance a two-part argument about the ways in which data infrastructures are transforming state-citizen relations through on the one hand advancing an actuarial logic based on personalised risk and the individualisation of social problems (what I refer to as responsibilisation) and, on the other, entrenching a dependency on an economic model that perpetuates the circulation of data accumulation (what I refer to as rentierism). These mechanisms, I argue, fundamentally shift the 'matrix of social power' that made the modern welfare state possible and position questions of data infrastructures as a core component of how we need to understand social change. "We are witnessing the gradual disappearance of the postwar British welfare stat