Designing Experimental Studies

In the last years, experiments became more and more widely applied - be it in academic research or A/B testing in companies. Due to their high internal validity, experiments are an important part of the methods ecosystem and researchers will benefit from integrating them into their methodological tool kit. This paper aims to summarize the most important content of the ICIS 2019 Professional Development Workshop. The workshop targets researchers with no or very basic training in experimental methods. It introduces the essentials of understanding and planning state-of-the-art experimental research and covers common pitfalls and challenges. Acknowledgment This work has been funded by the Federal Ministry of Education and Research of Germany (BMBF) under grant no. 16DII116 (“Deutsches Internet-Institut”)

Loot Box Purchase Decisions in Digital Business Models: The Role of Certainty and Loss Experience

Game providers are increasingly employing and selling loot boxes, which can be considered virtual goods that consist of further virtual goods on a randomized basis. As such, game providers can foster profitability without impeding user experience. Drawing on prospect theory, we investigate ideas for the design of loot box menus to optimize revenue generation and user well-being. By conducting a contest-based online experiment with 159 participants, our analyses reveal that including certain (vs. uncertain) content in loot boxes can influence users’ purchase behaviors and thus increase revenues. Moreover, this effect increases when participants previously experienced a loss. Thus, our findings demonstrate that game providers can profit from offering certain content in loot boxes

Why and How Online Experiments Can Benefit Information Systems Research

Online experiments have become an important methodology in the study of human behavior. While social scientists have been quick to capitalize on the benefits of online experiments, information systems (IS) researchers seem to be among the laggards in taking advantage of this emerging paradigm, despite having the research motivations and technological capabilities to be among the leaders. A major reason for this gap is probably the secondary role traditionally attributed in IS research to experimental methods, as repeatedly demonstrated in methodological reviews of work published in major IS publication outlets. The purpose of this editorial is to encourage IS researchers interested in online behavior to adopt online experiments as a primary methodology, which may substitute for traditional lab experiments and complement nonexperimental methods. This purpose is pursued by analyzing why IS research has lagged behind neighboring disciplines in adopting experimental methods, what IS research can benefit from utilizing online experiments, and how IS research can reap these benefits. The prescriptive analysis is structured around key considerations that should be taken into account in using online experiments to study online behavior

BLOCKCHAIN-BASED DATA SHARING SYSTEM: AN EXPERIMENTAL ANALYSIS OF BEHAVIOURAL FEATURES AFFECTING INTER-ORGANISATIONAL COOPERATION

This paper investigates the role of managers’ behavioural features on Blockchain Technology (BCT) appropriation within an online scenario-based behavioural experiment. At the intersection between Management of Information Systems and Experimental Economics, the scenario describes a BCT set-up as a new governance mechanism facilitating data sharing among organisations. The experiment involves BCT experts who performed two different Public Good Games (PGG): the first, reflecting a traditional data sharing system while the second was concerned with an exogenous minimum contribution level representing a BCT-based system. Results reveal that experts’ beliefs play a positive role in BCT early appropriation at an ecosystem level. When BCT-based data-sharing is established, and information about others’ cooperation is available, early appropriation level still affects the managers’ contribution and relative appropriation, moderating the role of behavioural features, such as beliefs and generalised trust

Privacy Risks in Digital Markets: The Impact of Ambiguity Attitudes on Transparency Choices

Transparency is viewed as an essential prerequisite for consumers to make informed privacy decisions in digital markets. However, it remains an open research question whether and when individuals actually prefer transparency about privacy risks when given a chance to avoid it. We investigate this question with a randomized controlled online experiment based on an Ellsberg-type design, where subjects repeatedly choose between risk and ambiguity while facing the threat of an actual disclosure of their personal data. We find empirical support for ambiguity attitudes as a novel behavioral mechanism underlying people\u27s transparency choices in privacy contexts. In particular, we find that most individuals avoid ambiguity and prefer transparency for low likelihood privacy losses. However, this pattern reverses for high likelihood losses and when subjects perceive data disclosure as a gain. Most notably, a significant share of people seek ambiguity and thus prefer to avoid transparency when facing high likelihood privacy risks

Promoting less complex and more honest price negotiations in the online used car market with authenticated data

Online peer-to-peer (P2P) sales of used and or high-value goods are gaining more and more relevance today. However, since potential buyers cannot physically examine product quality during online sales, information asymmetries and consequently uncertainty and mistrust that already exist in offline sales are exacerbated in online markets. Authenticated data platforms have been proposed to solve these problems by providing authenticated data about the negotiation object, integrating it into text-based channels secured by IT. Yet, we know little about the dynamics of online negotiations today and the impact of the introduction of authenticated data on online negotiation behaviors. We address this research gap based on two experimental studies along with the example of online used-car trade. We analyze users’ communicative and strategic actions in current P2P chat-based negotiations and examine how the introduction of authenticated data affects these behaviors using a conceptional model derived from literature. Our results show that authenticated data can promote less complex negotiation processes and more honest communication behavior between buyers and sellers. Further, the results indicate that chats with the availability of authenticated data can positively impact markets with information asymmetries. These insights provide valuable contributions for academics interested in the dynamics of online negotiations and the effects of authenticated data in text-based online negotiations. In addition, providers of trade platforms who aim to advance their P2P sales platforms benefit by achieving a competitive advantage and a higher number of customers

A Market-Based Approach to Facilitate the Organizational Adoption of Software Component Reuse Strategies

Despite the theoretical benefits of software component reuse (and the abundance of component-based software development on the vendor side), the adoption of component reuse strategies at the organizational level (on the client side) remains low in practice. According to research, the main barrier to advancing component-based reuse strategies into a robust industrial process is coordination failures between software producers and their customers, which result in high acquisition costs for customers. We introduce a component reuse licensing model and combine it with a dynamic price discovery mechanism to better coordinate producers’ capabilities and customer needs. Using an economic experiment with 28 IT professionals, we investigate the extent to which organizations may be able to leverage component reuse for performance improvements. Our findings suggest that implementing component reuse can assist organizations in addressing the issue of coordination failure with software producers while also lowering acquisition costs. We argue that similar designs can be deployed in practice and deliver benefits to software development in organizations and the software industry

Multi-party certification on blockchain and its impact in the market for lemons

Markets in which similar goods of different qualities are sold suffer from information asymmetries and their negative consequences. Dealers have established themselves, and mediate these markets through their use of quality signals. While these signals help to mitigate information asymmetries, these markets still function well below their optimum: a large share of goods sold are overpriced, and most of the benefits are reaped by intermediaries. In this paper we build on prior research that proposes the use of blockchain as an enabler for trusted, decentralized asset documentation. Applying a socio-technical lens, we describe how blockchain-enabled multi-party certification affords dealers the action potential to send signals that are more closely correlated to the unobservable quality of the underlying good (i.e., signals with a higher fit) than the signals they send today. We then both theorize and experimentally explore the market effects of the two types of signals. Using data from a laboratory market experiment with 210 participants, we find empirical evidence that multi-party certification affords dealers the action potential to send signals of significantly higher fit than those sent by intermediaries alone, leading to a reduction in information asymmetries, a more efficient allocation of goods, and an increase in market fairness

Integrating Explanatory/Predictive and Prescriptive Science in Information Systems Research

The scholarly information systems (IS) field has a dual role. As an explanatory and predictive science, the field contributes to explaining the pervasive IS that shape the digital age and sometimes also makes predictions about those phenomena. As a prescriptive science, it participates in creating IS-related innovations by identifying means-ends relationships. The two can beneficially interact, such as when explanatory theory provides the basis for generating prescriptions or when applicable knowledge produces explanatory insights. In this commentary, we contribute to integrating these two roles by proposing a framework to help IS researchers navigate the field’s duality to extend the cumulative scholarly knowledge that it creates in terms of justified explanations and predictions and justified prescriptions. The process we describe builds on ongoing, dynamic, iterative, and interrelated research cycles. We identify a set of integrative research practices that occur at the interface between explanatory and predictive science and prescriptive science—the explanation-prescription nexus. We derive guidelines for IS research

Organizational Decision-making in the Age of Big Data and Artificial Intelligence

This dissertation examines organizational decision-making in the context of big data and artificial intelligence (machine learning) technologies. There are three studies. All three focus on collaborative decision-making in organizations, with study 1 examining it in the context of big data, while study 2 and 3 in the context of artificial intelligence. Study 1: This study examines the impact of different manners of presenting information on collaborative decision-making performance. Using controlled economic experiments, I assign participants with a resource allocation decision-making task (adapted from the game theoretic public goods provision problem) and examine the collaborative outcomes of groups when exposed to different levels of information aggregation and visualization formats. Interestingly, the results show that in certain cases, the more effective means of presenting information for individuals (i.e., graphs or tables compared to raw data) do not bode as well for groups. This study contributes to the information visualization literature, which has mostly looked at individual decision-making, by examining the collaborative task context and by combining perspectives from experimental game theory, cognitive fit and information processing theories. Methodologically, this study also contributes to the information visualization literature by accounting for the dynamics of collaborative decision-making over time. Study 2: Organizations increasingly deploy artificial intelligence (AI) systems to automate specific tasks and assist human experts in organizational decision-making. In this study, I focus on complex task settings wherein human decision-makers work with AI systems. Using credit authorization for consumer loans as our specific context, I conduct an economic experiment with a repeated round design to investigate how organizations can create business value from the new human–machine collaborative decision-making paradigm. This study contributes to extant literatures on algorithmic decision-making and automation by moving beyond only examining individual decision-makers’ attributes to examine the intertwined roles of organizational factors and AI’s characteristics. The results show that when firms implement complementary organizational practices in parallel with AI investments, they achieve higher levels of algorithm appreciation, leading to better decisions, made with stronger confidence, in turn increasing organizational profits. I also show that human decision-makers and machines develop increasingly more effective work relationships over time and outperform AI machines in stand-alone settings. Finally, I show evidence that keeping humans in the loop could enable AI-powered firms to achieve the most productive outcomes. Study 3: Extant research on algorithmic bias has mostly approached the subject from a technical perspective, with few studies investigating the decision bias of human-machine collaborative decision-making, wherein human experts have the final say after working with the algorithms. In this study, I conduct a controlled economic experiment with a repeated-round design. I assign participants with a task that models a complex organizational decision-making process wherein human decision-makers (DMs) work with an AI repeatedly over 10 decision periods to evaluate consumer loan applications. I use loan data from a large-scale, historic dataset and manipulate the AI predictions to create two experimental conditions: (1) Prediction Bias, where DMs work with AI predictions that discriminate against one group of loan applicants and favor another, and (2) No Bias, where DMs work with AI predictions that treat the two loan applicant groups equally. This study contributes to current research on algorithmic bias mitigation and bias in human-machine collaboration by showing that human DMs can over time learn to adapt to a biased algorithm, implicitly detect the bias in the AI, adjust their behavior to significantly improve their performance, and importantly, outperform the biased AI working alone, both in terms of reducing decision bias and increasing organizational profit