Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis

In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the effects of scientists’ gender and professional status on verbosity during group meetings. Not only the analytic approach but also the operationalizations of key variables were left unconstrained and up to individual analysts. For instance, analysts could choose to operationalize status as job title, institutional ranking, citation counts, or some combination. To maximize transparency regarding the process by which analytic choices are made, the analysts used a platform we developed called DataExplained to justify both preferred and rejected analytic paths in real time. Analyses lacking sufficient detail, reproducible code, or with statistical errors were excluded, resulting in 29 analyses in the final sample. Researchers reported radically different analyses and dispersed empirical outcomes, in a number of cases obtaining significant effects in opposite directions for the same research question. A Boba multiverse analysis demonstrates that decisions about how to operationalize variables explain variability in outcomes above and beyond statistical choices (e.g., covariates). Subjective researcher decisions play a critical role in driving the reported empirical results, underscoring the need for open data, systematic robustness checks, and transparency regarding both analytic paths taken and not taken. Implications for organizations and leaders, whose decision making relies in part on scientific findings, consulting reports, and internal analyses by data scientists, are discussed

A change navigation-based, scenario planning process within a developing world context from an Afro-centric leadership perspective

Orientation: In the hyper turbulent context faced currently by organisations, more flexible strategic planning approaches, such as scenario planning which take into account a more comprehensive range of possible futures for an organisation, will position organisations better than conventional forecast and estimates that depend only on a single, linearly extrapolated, strategic response. Research purpose: This study aimed to investigate how scenario-based planning (a strictly cognitive management tool) can be combined with organisational change navigation (a practice addressing the emotionality of change) and how this integrated process should be aligned with the prerequisites imposed by a developing country context and an Afro-centric leadership perspective in order to make the process more context relevant and aligned. Motivation for the study: The integration of organisational change navigation with conventional scenario based planning, as well as the incorporation of the perquisites of a developing countries and an Afro-centric leadership perspective, will give organisations a more robust, holistic strategic management tool that will add significantly more value within a rapidly, radically and unpredictably changing world. Research design, approach and method: The adopted research approach comprised a combination of the sourcing of the latest thinking in the literature (the ‘theory’) as well as the views of seasoned practitioners of scenario planning (the ‘practice’) through an iterative research process, moving between theory and practice, back to practice and finally returning to theory in order to arrive at a validated expanded and enhanced scenario-based planning process which is both theory and practice ‘proof’. Main findings: A management tool incorporating the change navigation and the unique features of developing countries and Afro-centric leadership was formulated and empirically validated. This management tool is referred to as a change navigation based, scenario planning process (CNBSPP). Practical/managerial implications: CNBSPP is available for use by organisations wishing to apply a strategic planning tool that fits within a developing country context and an Afro-centric leadership approach. Contribution/value add: The research makes a unique contribution to the current level of knowledge by integrating two disciplines usually practised independently of one another, namely scenario-based planning and organisational change navigation. It also embedded the process into a different context of application, that is, the developed world as viewed from an Afro-centric leadership perspective

Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis

In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the effects of scientists’ gender and professional status on verbosity during group meetings. Not only the analytic approach but also the operationalizations of key variables were left unconstrained and up to individual analysts. For instance, analysts could choose to operationalize status as job title, institutional ranking, citation counts, or some combination. To maximize transparency regarding the process by which analytic choices are made, the analysts used a platform we developed called DataExplained to justify both preferred and rejected analytic paths in real time. Analyses lacking sufficient detail, reproducible code, or with statistical errors were excluded, resulting in 29 analyses in the final sample. Researchers reported radically different analyses and dispersed empirical outcomes, in a number of cases obtaining significant effects in opposite directions for the same research question. A Boba multiverse analysis demonstrates that decisions about how to operationalize variables explain variability in outcomes above and beyond statistical choices (e.g., covariates). Subjective researcher decisions play a critical role in driving the reported empirical results, underscoring the need for open data, systematic robustness checks, and transparency regarding both analytic paths taken and not taken. Implications for organizations and leaders, whose decision making relies in part on scientific findings, consulting reports, and internal analyses by data scientists, are discussed