National audienceBased on the experience gained on the observation of the corpora developement in HLT, the authors want to warn the Big Data community about some recent usage of hu-man computation. For instance, the growing use in the HLT community of crowdsourcing methods, and especially of microworking retributed crowsourcing platforms, lead to many ethical, economical and juridical concerns. The au-thors want also to foster some behaviours, especially con-cerning traceability, implemented in the form of a charter, the Ethics and Big Data Charter 

Adda, Gilles

Besacier, Laurent

Couillault, Alain

De Mazancourt, Hugues

Fort, Karen

Mariani, Joseph

de Mazancourt, Hugues

Hal - Université Grenoble Alpes

”Where the data are coming from?” Ethics,crowdsourcing and traceability for Big Data in HumanLanguage TechnologyGilles Adda, Laurent Besacier, Alain Couillault, Kare¨n Fort, Joseph Mariani,Hugues De MazancourtTo cite this version:Gilles Adda, Laurent Besacier, Alain Couillault, Kare¨n Fort, Joseph Mariani, et al.. ”Wherethe data are coming from?” Ethics, crowdsourcing and traceability for Big Data in HumanLanguage Technology. Crowdsourcing and human computation multidisciplinary workshop,Sep 2014, Paris, France. 2014. <hal-01078045>HAL Id: hal-01078045https://hal.archives-ouvertes.fr/hal-01078045Submitted on 28 Oct 2014HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestine´e au de´poˆt et a` la diffusion de documentsscientifiques de niveau recherche, publie´s ou non,e´manant des e´tablissements d’enseignement et derecherche franc¸ais ou e´trangers, des laboratoirespublics ou prive´s."Where the data are coming from?" Ethics, crowdsourcingand traceability for Big Datain Human Language TechnologyGilles Adda∗LIMSI and IMMI-CNRSRue John von Neumann91405 Orsay cedex Franceadda@immi-labs.orgLaurent BesacierLaboratoire d’Informatique deGrenobleUJF BP5338041 Grenoble Cedex 9,Francelaurent.besacier@imag.frAlain CouillaultUniversité de La Rochelle/L3IAv. Michel Crépeau17042 La Rochelle Cedex 01,Francealain.couillault@univ-lr.frKarën FortUniversité de Lorraine/LORIA54500 Vandœuvre-lès-Nancy,Francekaren.fort@loria.frJoseph J. MarianiLIMSI and IMMI-CNRSRue John von Neumann91405 Orsay cedex Francejoseph.mariani@limsi.frHugues de MazancourtEptica-Lingwayhugues.de-mazancourt@eptica.comABSTRACTBased on the experience gained on the observation of thecorpora developement in HLT, the authors want to warnthe Big Data community about some recent usage of hu-man computation. For instance, the growing use in theHLT community of crowdsourcing methods, and especiallyof microworking retributed crowsourcing platforms, lead tomany ethical, economical and juridical concerns. The au-thors want also to foster some behaviours, especially con-cerning traceability, implemented in the form of a charter,the Ethics and Big Data Charter .KeywordsEthics, crowdsourcing, traceability1. INTRODUCTIONAt the turn of the 90s, Human Language Technology (HLT)changed drastically, with the developpement of probabilis-tic machine learning methods and the establishment of theevaluation paradigm. One crucial result implied by thesechanges is the development of corpora of growing sizes andof varying natures. From speech corpora of few hours andtext corpora of few millions of words, we developed our mod-els from Multimedia, Multimodal, Multilingual (3M) datacontaining for instance thousands of hours of video and ter-awords text corpora coming from social media, newspapers,subtitles, transcriptions, translations, etc.∗Alphabetic orderMASTODONS Paris, FranceTo be able to manage the constitution and the distribu-tion of these corpora, specific agencies have been developedwithin the HLT community, such as LDC1 or ELDA.2 Butat a certain point, the growth of the needed corpora alongthe famous Big Data 3Vs3 axes, goes beyond the classicalwell-controlled developement of corpora, relying on a metic-ulous and safe selection of data, and retributed employeesto annotate them. We did see more and more corpora de-sign involving the use of crowdsourcing, and especially theuse of microworking, monetary compensated work platformsuch as Amazon Mechanical Turk. The authors, as academicand industrial HLT actors, wrote numerous positions papersto warn the community on ethical, economical and juridicalconcerns about mturk [1, 7, 2, 8]. From these statements,we found that retributed crowdsourcing was only one (im-portant) symptom of way of doing in the HLT communityfor the constitution and annotation of corpora.Some of the authors have designed the Ethics and Big DataCharter [4] in collaboration with representatives from inter-est groups, private companies and academic organizations,including ELDA, ATALA,4 AFCP5 and APROGED.6 Thepurpose of this Charter is to provide resources developers,not only in the HLT community, but more broadly in theBig Data community, with a precise framework to docu-ment their resources and ensure their traceability and trans-parency.1Linguistic Data Consortium, http://www.ldc.upenn.edu/2Evaluations and Language resources Distribution Agency,http://www.elda.org/3Volume, Velocity, Variety4Association pour le Traitement Automatique des Langueshttp://www.atala.org5Association Franc¸aise de Communication Parle´e http://www.afcp-parole.org6Association de la Maˆıtrise et de la Valorisation des contenushttp://www.aproged.org2. TOWARDSTRACEABILITY: THEETHICSAND BIG DATA CHARTERTo adopt an ethical behavior in developing, funding, using orpromoting language resources is first and above all a matterof choice: for the provider, deciding which approach to adopt– crowdsourcing or not –, or which platform to request on,or the level of remuneration of the workers; for the fundingagency, choosing which project to fund; for users, choosingwhich resource to use or acquire.2.1 Contents of the CharterThe Ethics and Big Data Charter is provided as a form to befilled in by the dataset provider. It is split into three majorsections: traceability, intellectual property and specific legis-lation, preceded by a short identification section containingthe names of the resource, the contact and responsible per-sons and a short description of the data set. We describebelow some of the important points listed in the charter.2.1.1 TraceabilityTraceability is key to our purpose of putting forward ethicalissues. The traceability part of the charter allows to pre-cise the relationship between the resource provider and theworkers involved in developing the resource, including legalbounding, workers skills, selection criteria.Specific focus is put on personal data, i.e. data such as voiceor video recording, which can provide a means to identify aperson directly or indirectly. The Charter requires to preciseif and how the data is de-identified, and if and how the indi-viduals were informed of the purpose of the data collection.Quality assurance is another major aspect of traceability ad-dressed by the charter, as it requires to document the qualityassurance strategy, so that the user of the data set is fullyinformed on the level of quality s/he can expect.2.1.2 License and copyrightThanks to a great deal of effort accomplished in the defini-tion of – mainly open source – license schemes, it has becomecommon practice to attach a license to a data set. The Li-cense and Copyright section of the Charter goes beyond thisand puts the focus on questions which may be disregarded,like ensuring that the legal or moral copyrights of the per-sons who worked on compiling, enriching or transformingthe data are respected.2.1.3 Specific legal requirementsA third section of the Ethics and Big Data Charter dealswith legal requirements that may arise from certain proper-ties of the data set. For example, a country may have issuedspecific laws regarding the storing, use and/or dissemina-tion of personal data. The Charter serves as a reminder forchecking if such requirements exist.2.2 AvailabilityThe Ethics and Big Data Charter is available on-line.7 Ex-amples of charters are also provided.3. CONCLUSION AND PERSPECTIVESIn [5], we showed that even the most widely referred to lan-guage resources present documentation lacks, in particular7http://wiki.ethique-big-data.orgregarding the persons who produced the work and which ex-ternal resources have been used. We assume that this alsoapplies for many corpora used in the Big Data community.The reality we are facing now in the call for projects of allthe national and international funding research agencies isa growing interest for Big Data, and a foreseeable burst inthe number of Big Data related projects, with the use, forinstance, of personal data from Twitter or Facebook, thedevelopment of crowdsourcing and so on. But we also knowthat for these data, uncertainties about privacy, the waythey have been acquired, who has really worked, is likely tobe of an order of magnitude more important than for thepresent resources.Other initiatives and articles have tackled this subject of Bigdata and Ethics (see for instance [6, 3, 9]), most of themdealing with the problem of privacy and surveillance.For these reasons, we think it is crucial to gather all theinitiatives such as the Ethics and Big Data Charter whichaims at promoting ethics and traceability in resources, inorder to propose, at the international level, a way to limitthe risks for all the actors (funding agencies, research labo-ratories, private companies) of the data added value chain,regarding the use of Big Data and bring the benefits of theEthics and Big Data Charter to current projects.4. REFERENCES[1] G. Adda and J. Mariani. Language resources andamazon mechanical turk: legal, ethical and other issues.In LISLR 2010, “Legal Issues for Sharing LanguageResources workshop”, LREC 2010, Valletta, Malta,May 2010.[2] G. Adda, J. Mariani, L. Besacier, and H. Gelas.Crowdsourcing for Speech Processing, chapter Economicand Ethical Background of Crowdsourcing for Speech,pages 303–334. Wiley, 2013.[3] D. Boyd and K. Crawford. Critical questions for bigdata. 5(15):662–679, May 2012.[4] A. Couillault and K. Fort. Charte E´thique et Big Data: parce que mon corpus le vaut bien ! In Linguistique,Langues et Parole : Statuts, Usages et Me´susages,Strasburg, France, July 2013. 4 pages.[5] A. Couillault, K. Fort, G. Adda, andH. De Mazancourt. Evaluating Corpora Documentationwith regards to the Ethics and Big Data Charter. InInternational Conference on Language Resources andEvaluation (LREC), Reykjavik,Iceland, May 2014.[6] K. Davis. Ethics of Big Data: Balancing Risk andInnovation. O’Reilly Media, Inc., 2012.[7] K. Fort, G. Adda, and K. B. Cohen. Amazonmechanical turk: Gold mine or coal mine?Computational Linguistics (editorial), 37(2), 2011.[8] K. Fort, G. Adda, B. Sagot, J. Mariani, andA. Couillault. HLT Challenges for Computer Scienceand Linguistics, chapter Crowdsourcing for LanguageResource Development: Criticisms about AmazonMechanical Turk Overpowering Use. Springer, 2014.[9] N. M. Richards and J. H. King. Big Data Ethics. InWake Forest Law Review, May 2014. Available atSSRN: http://ssrn.com/abstract=2384174.

"Where the data are coming from?" Ethics, crowdsourcing and traceability for Big Data in Human Language Technology

Abstract

Similar works

Full text

Available Versions

Hal - Université Grenoble Alpes