Ethics, Logs and Videotape: Ethics in Large Scale User Trials and User Generated Content. Workshop organized at CHI'11.

Mobile Life CentreMobile 2.

Implementing Ethics for a Mobile App Deployment

This paper discusses the ethical dimensions of a research project in which we deployed a personal tracking app on the Apple App Store and collected data from users with whom we had little or no direct contact. We describe the in-app functionality we created for supporting consent and withdrawal, our approach to privacy, our navigation of a formal ethical review, and navigation of the Apple approval process. We highlight two key issues for deployment-based research. Firstly, that it involves addressing multiple, sometimes conflicting ethical principles and guidelines. Secondly, that research ethics are not readily separable from design, but the two are enmeshed. As such, we argue that in-action and situational perspectives on research ethics are relevant to deployment-based research, even where the technology is relatively mundane. We also argue that it is desirable to produce and share relevant design knowledge and embed in-action and situational approaches in design activities

A practical framework for ethics - the PD-Net approach to supporting ethics compliance in public display studies

Research involving public displays often faces the need to study the effects of a deployment in the wild. While many organizations have institutionalized processes for ensuring ethical compliance of such human subject experiments, these may fail to stimulate sufficient awareness for ethical issues among all project members. Some organizations even require such assessments only for medical research, leaving computer scientists without any incentive to consider and reflect on their study design and data collection practices. Faced with similar problems in the context of the EU-funded PD-Net project, we have implemented a step-by-step ethics process that aims at providing structured yet lightweight guidance to all project members, both stimulating the design of ethical user studies, as well as providing continuous documentation. This paper describes our process and reports on 3 years of experience using it. All materials are publicly available and we hope that other projects in the area of public displays, and beyond, will adopt them to suit their particular needs.EC -European Commission(244011

Towards a Taxonomy of Ethical Considerations in Crowdsourcing

Crowdsourcing is a growing industry, where millions of individuals and businesses have begun tapping into the crowd to perform work. Despite this growth, crowd work and labor contains few regulations. Previous researchers have highlighted examples of ethical challenges organizations and individuals face in crowdsourcing. This paper explores these challenges, using Value Sensitive Design and Transparency literature to identify candidate ethical principles in crowdsourcing. Integrating these principles with ethical dilemmas, crowdsourcing models, and affected stakeholders, this research uses a deductive approach to develop a taxonomic framework of ethical considerations in crowdsourcing. The resulting taxonomy provides practical and theoretical contributions. Organizations choosing to use crowdsourcing can refer to the classification to understand ethical implications, as well as accounting for ethical requirements in the design and governance of projects. Researchers can expand the classification to gain understanding of each element and the interrelationships. Finally, we describe specific directions for future research

Mass participation user trials

This thesis investigates how researchers can take advantage of the rapid adoption of mobile technology that has brought with it transformations in social and cultural practice; the expectations of what computers are, what they can do, and the role of digital objects in everyday life. In particular this thesis presents and discuses the use of new App Store style software distribution methods to reduce the cost, in terms of researcher time and hardware, of recruiting a large group of participants for a trial ‘in the wild’ while increasing the potential diversity of users is becoming an attractive option for researchers pursuing the ubicomp vision. It examines the procedures for running large scale trials with the deployment of three applications released to a combined user base of over 135,000 in such a way as to keep the qualitative detail necessary to inform design while gain- ing the diversity of users for claims of generalisability. More generally, it discusses the results that can be expected from this ‘mass participation’ approach, and the ethical responsibilities they place upon researchers. The contributions of this thesis for mobile HCI show that in large-scale trials, relatively rich qualitative data can be collected along with substantial quantitative data, and that a hybrid trial methodology combining a large- scale deployment with a local trial can be a powerful tool in addressing shortcomings of trials that are either solely local or solely global. This thesis also contributes guidelines for researchers running large-scale user trials that give consideration to the established research norms and practices, in an attempt to strike a new balance between invasiveness and utility

Practical, appropriate, empirically-validated guidelines for designing educational games

There has recently been a great deal of interest in the potential of computer games to function as innovative educational tools. However, there is very little evidence of games fulfilling that potential. Indeed, the process of merging the disparate goals of education and games design appears problematic, and there are currently no practical guidelines for how to do so in a coherent manner. In this paper, we describe the successful, empirically validated teaching methods developed by behavioural psychologists and point out how they are uniquely suited to take advantage of the benefits that games offer to education. We conclude by proposing some practical steps for designing educational games, based on the techniques of Applied Behaviour Analysis. It is intended that this paper can both focus educational games designers on the features of games that are genuinely useful for education, and also introduce a successful form of teaching that this audience may not yet be familiar with

Designing for quality in real-world mobile crowdsourcing systems

PhD ThesisCrowdsourcing has emerged as a popular means to collect and analyse data on a scale for problems that require human intelligence to resolve. Its prompt response and low cost have made it attractive to businesses and academic institutions. In response, various online crowdsourcing platforms, such as Amazon MTurk, Figure Eight and Prolific have successfully emerged to facilitate the entire crowdsourcing process. However, the quality of results has been a major concern in crowdsourcing literature. Previous work has identified various key factors that contribute to issues of quality and need to be addressed in order to produce high quality results. Crowd tasks design, in particular, is a major key factor that impacts the efficiency and effectiveness of crowd workers as well as the entire crowdsourcing process. This research investigates crowdsourcing task designs to collect and analyse two distinct types of data, and examines the value of creating high-quality crowdwork activities on new crowdsource enabled systems for end-users. The main contribution of this research includes 1) a set of guidelines for designing crowdsourcing tasks that support quality collection, analysis and translation of speech and eye tracking data in real-world scenarios; and 2) Crowdsourcing applications that capture real-world data and coordinate the entire crowdsourcing process to analyse and feed quality results back. Furthermore, this research proposes a new quality control method based on workers trust and self-verification. To achieve this, the research follows the case study approach with a focus on two real-world data collection and analysis case studies. The first case study, Speeching, explores real-world speech data collection, analysis, and feedback for people with speech disorder, particularly with Parkinson’s. The second case study, CrowdEyes, examines the development and use of a hybrid system combined of crowdsourcing and low-cost DIY mobile eye trackers for real-world visual data collection, analysis, and feedback. Both case studies have established the capability of crowdsourcing to obtain high quality responses comparable to that of an expert. The Speeching app, and the provision of feedback in particular were well perceived by the participants. This opens up new opportunities in digital health and wellbeing. Besides, the proposed crowd-powered eye tracker is fully functional under real-world settings. The results showed how this approach outperforms all current state-of-the-art algorithms under all conditions, which opens up the technology for wide variety of eye tracking applications in real-world settings

Ethics, logs and videotape [CHI Workshop organisers and extended abstract]

As new technologies are appropriated by researchers, the community must come to terms with the evolving ethical responsibilities we have towards participants. This workshop brings together researchers to discuss the ethical issues of running large-scale user trials, and to provide guidance for future research. Trials of the scale of 10s or 100s of thousands of participants offer great potential benefits in terms of attracting users from vastly different geographical and social contexts, but raise significant ethical challenges. The inability to ensure user understanding of the information required to provide informed consent and problems involved in making users aware of the implications of the information being collected all beg the question: how can researchers ethically take advantage of the opportunities these new technologies afford

Ethics, Logs and Videotape : Ethics in Large Scale User Trials and User Generated Content

As new technologies are appropriated by researchers, the community must come to terms with the evolving ethical responsibilities we have towards participants. This workshop brings together researchers to discuss the ethical issues of running large-scale user trials, and to provide guidance for future research. Trials of the scale of 10s or 100s of thousands of participants offer great potential benefits in terms of attracting users from vastly different geographical and social contexts, but raise significant ethical challenges. The inability to ensure user understanding of the information required to provide informed consent and problems involved in making users aware of the implications of the information being collected all beg the question: how can researchers ethically take advantage of the opportunities these new technologies afford