Match Made by Humans: A Critical Enquiry into Human-Machine Configurations in Data Labelling

We present a critical ethnographic study of data labelling conducted in a Bangalore-based AI start-up. Labelled datasets are primarily produced by human data workers. We explore how humans and machines are configured together in data labelling and what are the demands placed on human workers, including on their body and cognition, while being assigned in the service of machine intelligence. We also show how these human-machine configurations sustain and reproduce the seamless functioning of apparently “autonomous” AI as a normative vision. Though labelled datasets are an indispensable prerequisite to creating ML/AI-based systems, the human labour that produces these datasets cannot be acknowledged fully if the techno-entrepreneurial vision of “self-learning” machine intelligence is to be celebrated and sustained. In pursuit of a normative position of what AI should be, we are left with a denial of how AI is actually produced now

Making Data Work Count

In this paper, we examine the work of data annotation. Specifically, we focus on the role of counting or quantification in organising annotation work. Based on an ethnographic study of data annotation in two outsourcing centres in India, we observe that counting practices and its associated logics are an integral part of day-to-day annotation activities. In particular, we call attention to the presumption of total countability observed in annotation - the notion that everything, from tasks, datasets and deliverables, to workers, work time, quality and performance, can be managed by applying the logics of counting. To examine this, we draw on sociological and socio-technical scholarship on quantification and develop the lens of a 'regime of counting' that makes explicit the specific counts, practices, actors and structures that underpin the pervasive counting in annotation. We find that within the AI supply chain and data work, counting regimes aid the assertion of authority by the AI clients (also called requesters) over annotation processes, constituting them as reductive, standardised, and homogenous. We illustrate how this has implications for i) how annotation work and workers get valued, ii) the role human discretion plays in annotation, and iii) broader efforts to introduce accountable and more just practices in AI. Through these implications, we illustrate the limits of operating within the logic of total countability. Instead, we argue for a view of counting as partial - located in distinct geographies, shaped by specific interests and accountable in only limited ways. This, we propose, sets the stage for a fundamentally different orientation to counting and what counts in data annotation.Comment: Accepted for publication at CSCW 2024. Forthcoming in the Proceedings of the ACM on Human-Computer Interactio

Politics of Data in & as News: A Data Justice Perspective

In this paper, we examine the practice of data journalism in India to highlight the politics that constitute as well as arise from the use of data in and as news. Drawing on the framework of data justice, we challenge the normative claims of data neutrality and objectivity by underlining the essentially political nature of data and the larger politics that shape access, use and meanings of data. We highlight four ways in which data politics can be observed: (i) data as a source of credibility, (ii) politics of data access, (iii) politics of censorship, and (iv) data as a new site of polarization. We position our findings within the larger discourse on datafication of development to argue that data in and of itself neither leads to justice or injustice. Rather it is the politics of and around data that unravels its actual potential and its consequences for development