The Perils and Promises of Big Data Research in Information Systems

With the proliferation of “big data” and powerful analytical techniques, information systems (IS) researchers are increasingly engaged in what we label as big data research (BDR)—research based on large digital trace datasets and computationally intensive methods. The number of such research papers has been growing rapidly in the top IS journals during the last decade, with roughly 16% of papers in 2018 employing this approach. In this editorial, we propose five conjectures that articulate the potential consequences of increasing BDR prevalence for the IS field’s research goals and outputs. We discuss ways in which IS researchers may be able to better leverage big data and new analysis techniques to conduct more impactful research. Our intent with these conjectures and analyses is to stimulate debate in the IS community. Indeed, we need a productive discussion about how emerging new research methods, digital trace data, and the development of indigenous theory relate to and can support one another

DIGITAL TRACE DATA RESEARCH IN INFORMATION SYSTEMS: OPPORTUNITIES AND CHALLENGES

Digital trace data research is an emerging paradigm in Information Systems (IS). Whether for theory development or theory testing, IS scholars increasingly draw on data that are generated as actors use information technology. Because they are ‘digital’ in nature, these data are particularly suitable for computational analysis, i.e. analysis with the aid of algorithms. In turn, this opens up new possibilities for data analysis, such as process mining, text mining, and network analysis. At the same time, the increasing use of digital trace data for research purposes also raises questions and potential issues that the research community needs to address. For example, one key question is what constitutes a valid contribution to the body of knowledge and how digital trace data research influences our collective identity as a field? In this panel, we will discuss opportunities and challenges associated with digital trace data research. Reflecting on the panelists’ and the audience’s experience, we will point to strategies to mitigate common pitfalls and outline promising research avenues

How Information Systems can Support Heuristic Decision Making: A Pilot Study

To answer the long-standing question of how to construct information systems to support heuristic decision making, we propose a new model of decision support for tacit knowledge. The aim of proposing a new model of decision support for tacit knowledge is to advance a general model of how experts describe the tacit aspects of their decision making. Such a model has implications for the construction of explanations by artificial intelligence. In a pilot case study of shift work planning, we explore the use of heuristics by consultants to interpret the conditions, outputs, and quality of shift work and to generate recommendations for changes to the organisation of shift work in organisations. The proposed, full case study to follow this pilot study will evaluate a conceptual model of decision support to inform how heuristics can explain decisions made by human, and by extension, artificial intelligence agents

Barking Up the Wrong Tree? Reconsidering Policy Compliance as a Dependent Variable within Behavioral Cybersecurity Research

A rich body of research examines the cybersecurity behavior of employees, with a particular focus on explaining the reasons why employees comply with (or violate) organizational cybersecurity policies. However, we posit that this emphasis on policy compliance is susceptible to several notable limitations that could lead to inaccurate research conclusions. In this commentary, we examine the limitations of using cybersecurity policy compliance as a dependent variable by presenting three assertions: (1) the link between policy compliance and organizational-level outcomes is ambiguous; (2) policies vary widely in terms of their clarity and completeness; and (3) employees have an inconsistent familiarity with their own organization’s cybersecurity policies. Taken together, we suggest that studying compliance with cybersecurity policies reveals only a partial picture of employee behavior. In response, we offer recommendations for future research

Holding Space for Voices that Do Not Speak: Design Reform of Rating Systems for Platforms in GREAT Economies

Researchers can examine ethical implications of online rating systems to understand how they function as ‘knowledge instruments’ and affect social relations and networks connected with them. Research should address the fact that the underlying economic structures that design and deploy knowledge producing ‘technical objects’ on online platforms are not egalitarian and may create new circles of exclusion. Exploring implications of this for a starkly unequal country like India, we illustrate our ideas by integrating induction and abduction to study rating systems on a pan-India food discovery and delivery platform. Rating systems are borrowed from WEIRD contexts and our findings imply that the instrument studied here is designed to hear only some of many voices. Consequently, they might be ‘institutionalizing’ knowledge that is problematic for GREAT domains in which they are imposed. We highlight the need for decolonization of research approaches for GREAT domains and critical research of technical knowledge objects

Rigor, Relevance, and Practical Significance: A Real-life Journey to Organizational Value

In this essay, we describe a research journey focusing on how to analyze mouse cursor movements, typing fidelity, and data from other human-computer interaction (HCI) devices to better understand the end-user online experience. We begin by defining organizational value and how it relates to other aspects that researchers use to assess academic research quality. We then describe and contrast our research journey by demonstrating key research milestones: from achieving statistical significance to achieving practical significance and, finally, to reaching relevance to practice. We then explain how we crossed the chasm between academic research and technology commercialization (i.e., the last research mile). We conclude by describing the process one can follow to develop an initial prototype—the minimal viable product (MVP)—and how demonstrations with potential customers provides continuous insight and validation for evolving the commercial product capabilities to meet constantly changing and evolving customer and industry needs

GROUNDED COMPUTATIONAL ANALYSIS: A HANDS-ON APPROACH TO ANALYSING DIGITAL INNOVATION

As socio-technical processes related to digital innovation are increasingly connected and distributed across geographical, organisational, and temporal boundaries, the methods we use to study them must be adapted to accommodate the greater detail and scope of the phenomenon. Specifically, there is a need to operationalise methods for generating inductive theory of distributed digital innovation from digital trace data. An emerging stream of IS research on computationally intensive inductive theorising lays the groundwork for such methods. This paper builds on this foundation to develop a hands-on approach to operationalising grounded theorizing in computational analysis of digital trace data. The paper first conceptualises trace data of digital innovation as a new research context before articulating an approach to operationalising grounded theory in computational analysis of digital innovation. The application of the grounded computational analysis approach is then briefly illustrated in the context of digital trace data from an online social network before possible directions for further research are laid out

STUDYING DYNAMICS AND CHANGE WITH DIGITAL TRACE DATA: A SYSTEMATIC LITERATURE REVIEW

Digital trace data offer promising opportunities to study dynamics and change of various socio-technical phenomena over time. While we see a surge of empirical and conceptual articles, we lack a systematic understanding of why, how, and when digital trace data are or can be used to study dynamics and change. In this article, we present the findings of a systematic literature review to uncover common approaches, motivations, findings, and general themes in the existing literature. We systematically reviewed 40 studies that were published in premium outlets in the information systems field. Our review sheds light on (1) underlying purposes of such studies, (2) utilized data sources, (3) research contexts, (4) socio-technical phenomena of interest, (5) applied analytical methods, and (6) measures that are being used. Building on our findings, we point to several implications for research and shed light on avenues to advance this field in the future

Opening the Black-Box of AI: Challenging Pattern Robustness and Improving Theorizing through Explainable AI Methods

Machine Learning (ML) algorithms, as approach to Artificial Intelligence (AI), show unprecedented analytical capabilities and tremendous potential for pattern detection in large data sets. Despite researchers showing great interest in these methodologies, ML remains largely underutilized, because the algorithms are a black-box, preventing the interpretation of learned models. Recent research on explainable artificial intelligence (XAI) sheds light on these models by allowing researchers to identify the main determinants of a prediction through post-hoc analyses. Thereby, XAI affords the opportunity to critically reflect on identified patterns, offering the opportunity to enhance decision making and theorizing based on these patterns. Based on two large and publicly available data sets, we show that different variables within the same data set can generate models with similar predictive accuracy. In exploring this issue, we develop guidelines and recommendations for the effective use of XAI in research and particularly for theorizing from identified patterns

Inside a Data Science Team: Data Crafting in Generating Strategic Value from Analytics

Current research agrees that the value of data lies in analytics that generate valuable insights for strategic purposes. However, little is known about how these insights are derived by data scientists. This research reports on the work of an embedded data science team at an organization striving to use people analytics to improve its strategic human resource management. We find that to create strategically valuable analytics, data scientists engage in data crafting, an approach to data science work that relies on broadcasting the potential value of data science towards the organization, cultivating a shared vision of value within the team, and creating value-adding data products with organizational customers. To do so, the team requires appropriate positioning and autonomy within the organization. Our findings have implications on understanding the role of data science teams and organizational data with respect to strategy, and practical insights for realizing strategic value from analytics