A Modeling Approach for Measuring the Performance of a Human-AI Collaborative Process

Despite the unabated growth of algorithmic decision-making in organizations, there is a growing consensus that numerous situations will continue to require humans in the loop. However, the blending of a formal machine and bounded human rationality also amplifies the risk of what is known as local rationality. Therefore, it is crucial, especially in a data-abundant environment that characterizes algorithmic decision-making, to devise means to assess performance holistically. In this paper, we propose a simulation-based model to address the current lack of research on quantifying algorithmic interventions in a broader organizational context. Our approach allows the combining of causal modeling and data science algorithms to represent decision settings involving a mix of machine and human rationality to measure performance. As a testbed, we consider the case of a fictitious company trying to improve its forecasting process with the help of a machine learning approach. The example demonstrates that a myopic assessment obscures problems that only a broader framing reveals. It highlights the value of a systems view since the effects of the interplay between human and algorithmic decisions can be largely unintuitive. Such a simulation-based approach can be an effective tool in efforts to delineate roles for humans and algorithms in hybrid contexts

A Measurement Model of Value of Data for Decision Making in the Digital Era

The file attached to this record is the author's final peer reviewed version.Despite burgeoning opportunities for data-driven decisions, research shows that decision-makers are failing to make sense of data within a broader context of organizational change which presents the following pertinent questions: 1) how can decision-makers measure the value of data by giving a holistic account? 2) how should the organization-specific blending of Machine and human rationality be factored in the measurement model? This study tackles these questions by proposing a novel approach that combines system dynamics and the ability to incorporate data science methods. In addition to a conceptual description, this paper also describes a feasibility test conducted on a small-scale prototype set in a supply-chain context. The results show that the use of sophisticated models that have a local scope ("locally rational") might have unintended global consequences. It underscores the need for a holistic model in the decision-makers' toolkit providing the ability to run targeted simulations to assess digital investments