The Use of Online Panel Data in Management Research: A Review and Recommendations

Management scholars have long depended on convenience samples to conduct research involving human participants. However, the past decade has seen an emergence of a new convenience sample: online panels and online panel participants. The data these participants provide—online panel data (OPD)—has been embraced by many management scholars owing to the numerous benefits it provides over “traditional” convenience samples. Despite those advantages, OPD has not been warmly received by all. Currently, there is a divide in the field over the appropriateness of OPD in management scholarship. Our review takes aim at the divide with the goal of providing a common understanding of OPD and its utility and providing recommendations regarding when and how to use OPD and how and where to publish it. To accomplish these goals, we inventoried and reviewed OPD use across 13 management journals spanning 2006 to 2017. Our search resulted in 804 OPD-based studies across 439 articles. Notably, our search also identified 26 online panel platforms (“brokers”) used to connect researchers with online panel participants. Importantly, we offer specific guidance to authors, reviewers, and editors, having implications for both micro and macro management scholars

Using Amazon Mechanical Turk and Other Compensated Crowd Sourcing Sites

Crowdsourcing is becoming recognized as a powerful tool that organizations can use in order to get work done, this by freelancers and non-employees. We conceptualize crowdsourcing as a subcategory of outsourcing, with compensated crowdsourcing representing situations in which individuals performing the work receive some sort of payment for accomplishing the organization’s tasks. Herein, we discuss how sites that create a crowd, such as Amazon Mechanical Turk, can be powerful tools for business purposes. We highlight the general features of crowdsourcing sites, offering examples drawn from current crowdsourcing sites. We then examine the wide range of tasks that can be accomplished through crowdsourcing sites. Large online worker community websites and forums have been created around such crowdsourcing sites, and we describe the functions they generally play for crowdsourced workers. We also describe how these functions offer opportunities and challenges for organizations. We close by discussing major considerations organizations need to take into account when trying to harness the power of the crowd through compensated crowdsourcing sites

A Study of Ethics in Crowd Work-Based Research

Crowd work as a form of a social-technical system has become a popular setting for conducting and distributing academic research. Crowd work platforms such as Amazon Mechanical Turk (MTurk) are widely used by academic researchers. Recent scholarship has highlighted the importance of ethical issues because they could affect the long-term development and application of crowd work in various fields such as the gig economy. However, little study or deliberation has been conducted on the ethical issues associated with academic research in this context. Current sources for ethical research practice, such as the Belmont Report, have not been examined thoroughly on how they should be applied to tackle the ethical issues in crowd work-based research such as those in data collection and usage. Hence, how crowd work-based research should be conducted to make it respectful, beneficent, and just is still an open question. This dissertation research has pursued this open question by interviewing 15 academic researchers and 17 IRB directors and analysts in terms of their perceptions and reflections on ethics in research on MTurk; meanwhile, it has analyzed 15 research guidelines and consent templates for research on MTurk and 14 published papers from the interviewed scholars. Based on analyzing these different sources of data, this dissertation research has identified three dimensions of ethics in crowd work-based research, including ethical issues in payment, data, and human subjects. This dissertation research also uncovered the “original sin” of these ethical issues and discussed its impact in academia, as well as the limitations of the Belmont Report and AoIR Ethical Guidelines 3.0 for Internet Research. The findings and implications of this research can help researchers and IRBs be more conscious about ethics in crowd work-based research and also inspire academic associations such as AoIR to develop ethical guidelines that can address these ethical issues

On the adoption of crowdsourcing for theory testing

This paper examines the possibilities of using the crowdsourcing strategy for theory testing. We first analyse the relationships between theory building and theory testing activities. Then, based on a systematic review of 248 papers published in MISQ, we characterise the intents and pat-tern systems of activities that have been used for theory testing. Finally, we ascertain which ac-tivities can be crowdsourced or not and pinpoint a set of pathways supporting partial and total crowdsourcing. The obtained results show that a large number of activities related to data gath-ering can be crowdsourced, and that a number of intents have viable pathways supporting par-tial crowdsourcing

Creation of Reliable Relevance Judgments in Information Retrieval Systems Evaluation Experimentation through Crowdsourcing: A Review

Test collection is used to evaluate the information retrieval systems in laboratory-based evaluation experimentation. In a classic setting, generating relevance judgments involves human assessors and is a costly and time consuming task. Researchers and practitioners are still being challenged in performing reliable and low-cost evaluation of retrieval systems. Crowdsourcing as a novel method of data acquisition is broadly used in many research fields. It has been proven that crowdsourcing is an inexpensive and quick solution as well as a reliable alternative for creating relevance judgments. One of the crowdsourcing applications in IR is to judge relevancy of query document pair. In order to have a successful crowdsourcing experiment, the relevance judgment tasks should be designed precisely to emphasize quality control. This paper is intended to explore different factors that have an influence on the accuracy of relevance judgments accomplished by workers and how to intensify the reliability of judgments in crowdsourcing experiment

EXAMINING THE UTILITY OF BEHAVIORAL ECONOMIC DEMAND IN ADDICTION SCIENCE

The marriage of perspectives from behavioral economic theory and learning theory has the potential to advance an understanding of substance use and substance use disorder. Behavioral economic demand is a central concept to this interdisciplinary approach. Evaluating demand in the laboratory and clinic can improve previous research on the relative reinforcing effects of drugs by accounting for the multi-dimensional nature of reinforcement rather than viewing reinforcement as a unitary construct. Recent advances in the commodity purchase task methodology have further simplified the measurement of demand values in human participants. This dissertation project presents a programmatic series of studies designed to demonstrate the utility of using a behavioral economic demand framework and the purchase task methodology for understanding substance use disorder through basic and applied science research. Experiments are presented spanning a continuum from theoretical and methodological development to longitudinal work and clinical application. These experiments demonstrate three key conclusions regarding behavioral economic demand. First, behavioral economic demand provides a reliable and valid measure of drug valuation that is applicable to varied drug types and participant populations. Second, behavioral economic demand is a stimulus-selective measure specifically reflecting valuation for the commodity under study. Third, behavioral economic demand provides incremental information about substance use in the laboratory and clinical setting above and beyond traditional measures of reinforcer valuation and other behavioral economic variables. These findings collectively highlight the benefits of behavioral economic demand and provide an important platform for future work in addiction science

Friendly Hackers to the Rescue: How Organizations Perceive Crowdsourced Vulnerability Discovery

Over the past years, crowdsourcing has increasingly been used for the discovery of vulnerabilities in software. While some organizations have extensively used crowdsourced vulnerability discovery, other organizations have been very hesitant in embracing this method. In this paper, we report the results of a qualitative study that reveals organizational concerns and fears in relation to crowdsourced vulnerability discovery. The study is based on 36 key informant interviews with various organizations. The study reveals a set of pre-adoption fears (i.e., lacking managerial expertise, low quality submissions, distrust in security professionals, cost escalation, lack of motivation of security professionals) as well as the post-adoption issues actually experienced. The study also identifies countermeasures that adopting organizations have used to mitigate fears and minimize issues. Implications for research and practice are discussed

Quality in Crowdsourcing - How software quality is ensured in software crowdsourcing

Crowdsourcing is a relatively new technique which aims to make a specific group of people contribute solutions to simple tasks or problems that are published online by some organization. For this they get some reward, which is usually economic in nature. This technique can be embraced by any kind of company, and since it is done online, it can turn out to be a bit problematic, especially when it comes to software development, because the whole process is out of the developing company’s hands. Some quality problems may arise during the process, such as a great amount of non-serious submissions and people presenting vague solutions because they are just trying to get the monetary reward. In order to make crowdsourcing successful these problems need to be solved, and companies which use this method for software development need to have some quality assurance for their products. This study tries to find out how companies using crowdsourcing deal with these problems and how they try to ensure some levels of quality in the final product. What we found is that companies embracing crowdsourcing use several methods in order to ensure a certain level of quality, such as rating, spam filters and reviews. There are many similarities in the underlying functions behind the methods each company uses such as motivating participants or finding the best solutions. These methods are applied at different stages throughout the crowdsourcing process. The exact relationships between the current use of these methods and the effect on software quality are not entirely apparent