Tragedy of the Commons - A Critical Study of Data Quality and Validity Issues in Crowd Work-Based Research

Crowd work platforms such as MTurk have been leveraged by academic scholars to conduct research and collect data. Though prior studies have discussed data quality and validity issues in crowd work via surveys and experiments, they kind of neglected to explore the scholars’ and particularly the IRB’s ethical concerns in these respects. In this study, we interviewed 17 scholars in six different disciplines and 15 IRB directors/analysts in the U.S. to fill this research gap. We identified common themes among our respondents but also discovered distinctive and even opposing views regarding the approval rate, rejection, internal and external research validity. Based on the findings, we discussed a potential Tragedy of the Commons regarding the data quality deterioration and the disciplinary differences regarding validity in crowd work-based research. Finally, we advocated that the IRB’s ethical concerns should be heard and respected

A Study of Ethics in Crowd Work-Based Research

Crowd work as a form of a social-technical system has become a popular setting for conducting and distributing academic research. Crowd work platforms such as Amazon Mechanical Turk (MTurk) are widely used by academic researchers. Recent scholarship has highlighted the importance of ethical issues because they could affect the long-term development and application of crowd work in various fields such as the gig economy. However, little study or deliberation has been conducted on the ethical issues associated with academic research in this context. Current sources for ethical research practice, such as the Belmont Report, have not been examined thoroughly on how they should be applied to tackle the ethical issues in crowd work-based research such as those in data collection and usage. Hence, how crowd work-based research should be conducted to make it respectful, beneficent, and just is still an open question. This dissertation research has pursued this open question by interviewing 15 academic researchers and 17 IRB directors and analysts in terms of their perceptions and reflections on ethics in research on MTurk; meanwhile, it has analyzed 15 research guidelines and consent templates for research on MTurk and 14 published papers from the interviewed scholars. Based on analyzing these different sources of data, this dissertation research has identified three dimensions of ethics in crowd work-based research, including ethical issues in payment, data, and human subjects. This dissertation research also uncovered the “original sin” of these ethical issues and discussed its impact in academia, as well as the limitations of the Belmont Report and AoIR Ethical Guidelines 3.0 for Internet Research. The findings and implications of this research can help researchers and IRBs be more conscious about ethics in crowd work-based research and also inspire academic associations such as AoIR to develop ethical guidelines that can address these ethical issues

“You Don’t Know Where It Will Stop” -- An Inquiry into Smartphone Users' Privacy Mental Models of Contextual Integrity

The Contextual Integrity (CI) theory provides a benchmark for privacy protection or violation according to the appropriateness of information collection and flows in a certain context. As privacy threats and protections develop and vie in various mobile contexts, how smartphone users represent the benchmark CI in their minds deserves exploration. In this study, we inquired into 18 smartphone users’ privacy mental models of CI. We found that they verbalized and visualized three patterns of information flow (i.e., unidirectional lines, branching tree, and complex network) and two categories of information collection (i.e., monetization-oriented and monitoring-based). With these mental models, our participants expressed numerous privacy concerns, such as unstoppable information sharing, data monetization, and surveillance. We discussed these findings and concluded that even though mobile operating systems and apps have claimed to be privacy-friendly and protective, some users remain dubious about such claims even though their privacy mental models may not accurately reflect reality

Designing a Mobile Crowdsourcing System for Campus Safety

Safety on college campuses remains a dire issue. Current reporting methods are still cumbersome and include no enhancing social aspect. Given the unique opportunity that universities are required to publish their safety report logs, we conducted a preliminary data analysis of a university's safety report log. The analysis allowed us to detect relevant trends in reporting behavior, specifically pertaining to where, when, and how soon the community would report safety incidents. Motivated by these findings and by literature promoting interactive reporting systems, we designed a novel mobile app which aims to enable the spread of crowdsourced public safety information. This app allows for immediate mass sharing of self-reported safety incidents, as well as the opportunity for witness reporting. Feedback from a paper prototype interview study indicated that these qualities would facilitate increased interactivity among its user community, and ultimately promote awareness of campus safety.ye

TRACE: A Stigmergic Crowdsourcing Platform for Intelligence Analysis

Crowdsourcing has become a frequently adopted approach to solving various tasks from conducting surveys to designing products. In the field of reasoning-support, however, crowdsourcing-related research and application have not been extensively implemented. Reasoning-support is essential in intelligence analysis to help analysts mitigate various cognitive biases, enhance deliberation, and improve report writing. In this paper, we propose a novel approach to designing a crowdsourcing platform that facilitates stigmergic coordination, awareness, and communication for intelligence analysis. We have partly materialized our proposal in the form of a crowdsourcing system which supports intelligence analysis: TRACE (Trackable Reasoning and Analysis for Collaboration and Evaluation). We introduce several stigmergic approaches integrated into TRACE and discuss the potential experimentation of these approaches. We also explain the design implications for further development of TRACE and similar crowdsourcing systems to support reasoning

Effects of allyl isothiocyanate fumigation on medicinal plant root knot disease control, plant survival, and the soil bacterial community

Abstract Background Allyl isothiocyanate (AITC) is a natural product with high volatility that is used as a biofumigant to alleviate soil-borne plant diseases, and problems such as root knot nematodes (RKNs) that necessitate continuous cropping. However, little research has assessed the effects of AITC fumigation on medicinal plants. Results AITC significantly reduced the population of RKNs in soil (p < 0.0001) and showed an excellent RKN disease control effect within 6 months after sowing Panax notoginseng (p < 0.0001). The seedling survival rate of 2-year-old P. notoginseng was approximately 1.7-fold higher after soil treatment with AITC (p = 0.1008). 16S rRNA sequencing indicated that the AITC treatment affected bacterial richness rather than diversity in consecutively cultivated (CC) soil. Furthermore, biomarkers with statistical differences between AITC-treated and untreated CC soil showed that Pirellulales (order), Pirellulaceae (family), Pseudomonadaceae (family), and Pseudomonas (genus) played important roles in the AITC-treated group. In addition, the microbiome functional phenotypes predicted using the BugBase tool suggested that AITC treatment is more conducive to improving CC soil through changes in the bacterial community structure. Crucially, our research also suggested that AITC soil treatment significantly increases soil organic matter (p = 0.0055), total nitrogen (p = 0.0054), and available potassium (p = 0.0373), which promotes the survival of a succeeding medicinal plant (Polygonatum kingianum). Conclusion AITC is an ecologically friendly soil treatment that affects the top 10 bacterial richness but not diversity. It could also provide a basis for a useful agricultural soil management measure to alleviate soil sickness

AOIR ETHICS PANEL 2: PLATFORM CHALLENGES

This panel is one of two sessions organized by the AoIR Ethics Working Committee. It collects five papers exploring a broad (but in many ways common) set of ethical dilemmas faced by researchers engaged with specific platforms such as Reddit, Amazon’s Mechanical Turk, and private messaging platforms. These include: a study of people's online conversations about health matters on Reddit in support of a proposed situated ethics framework for researchers working with publicly available data; an exploration into sourcing practices among Reddit researchers to determine if their sources could be unmasked and located in Reddit archives; a broader systematic review of over 700 research studies that used Reddit data to assess the kinds of analysis and methods researchers are engaging in as well as any ethical considerations that emerge when researching Reddit; a critical examination of the use of Amazon’s Mechanical Turk for academic research; and an investigation into current practices and ethical dilemmas faced when researching closed messaging applications and their users. Taken together, these papers illuminate emerging ethical dilemmas facing researchers when investigating novel platforms and user communities; challenges often not fully addressed–if even contemplated–in existing ethical guidelines. These papers are among those under consideration for publication in a special issue of the Journal of Information, Communication and Ethics in Society associated with the AoIR Ethics Working Committee and AoIR2021