(Re)considering the Concept of Literature Review Reproducibility

Literature reviews play a key role in academic research by describing, understanding, explaining, and testing the constructs and theories within a particular topic area. In recent years, various commentaries, debates, and editorials in the information systems (IS) field’s top journals have highlighted the importance of a trustworthy literature review process, including detailed discussions on systematicity and transparency. Although the reproducibility of a literature review has also been noted as important, it remains less recognized because of several terminology-related issues. This ambiguity could result in misunderstandings regarding the degree of trust that should be placed in a literature review’s process. In this research essay, we seek to clarify what makes a literature review reproducible, how it is distinct from related concepts, and when achieving it is desirable and feasible. We propose a series of clarifications and remedies to assist scholars within and outside the IS field in the preparation of stand-alone reviews

Towards Identifying and closing Gaps in Assurance of autonomous Road vehicleS - a collection of Technical Notes Part 1

This report provides an introduction and overview of the Technical Topic Notes (TTNs) produced in the Towards Identifying and closing Gaps in Assurance of autonomous Road vehicleS (Tigars) project. These notes aim to support the development and evaluation of autonomous vehicles. Part 1 addresses: Assurance-overview and issues, Resilience and Safety Requirements, Open Systems Perspective and Formal Verification and Static Analysis of ML Systems. Part 2: Simulation and Dynamic Testing, Defence in Depth and Diversity, Security-Informed Safety Analysis, Standards and Guidelines

Exploring Government Contractor Experiences Assessing and Reporting Software Development Status

Reports from academic, commercial, and government organizations have documented software-intensive system cost and schedule overruns for decades. These reports have identified lack of management insight into the software development process as one of many contributing factors. Multiple management mechanisms exist. However, these mechanisms do not support the assessment, and subsequent reporting, of software completion status. Additionally, the conceptual framework, based on industry standards, is limited in its relevance to this study due to an emphasis on what is needed while deferring implementation details. The purpose of this phenomenological study was to explore U.S. government contractors\u27 lived experiences of assessing and reporting software completion status with current measurement mechanisms. Twenty program or project managers responded to interview questions targeting positive and challenging experiences with current measurement mechanisms. Qualitative analysis of the experiential data was based on open and axial coding conducted on interview transcripts. Analysis indicated that costly resources are applied to metrics that do not provide the required level of management insight into completion status. These findings have positive social change implications for program managers, project managers, and researchers by documenting the need to develop relevant and cost-efficient status metrics to provide the critical insight required by management to reduce overruns

Measuring Trustworthiness of AI Systems: A Holistic Maturity Model

Artificial intelligence (AI) has an impact on business and society at large while posing challenges and risks. For AI adoption, trustworthiness is paramount, yet there appears to be a gap between theory and practice. Organizations need guidance in quantitatively assessing and improving the trustworthiness of AI systems. To address such challenges, maturity models have shown to be a valuable instrument. However, recent AI maturity models address trustworthiness only at the maturest level. As a response, we propose a model to integrate the concept of trustworthiness across the AI lifecycle management. In doing so, we follow Design Science Research to develop a holistic model highlighting the importance of trustworthiness throughout the AI adoption journey to realize the real value potential. This research-in-progress contributes to the emerging research on human-AI systems and managing AI. Our objective is to use the model for assessing, evaluating, and improving trustworthy AI on an organizational level

Psychometric properties of the TACT framework—Determining rigor in qualitative research

Introduction: The credibility of qualitative research has long been debated, with critics emphasizing the lack of rigor and the challenges of demonstrating it. In qualitative research, rigor encompasses explicit, detailed descriptions of various research stages, including problem framing, study design, data collection, analysis, and reporting. The diversity inherent in qualitative research, originating from various beliefs and paradigms, challenges establishing universal guidelines for determining its rigor. Additionally, researchers' often unrecorded thought processes in qualitative studies further complicate the assessment of research quality.Methods:To address these concerns, this article builds on the TACT framework, which was developed to teach postgraduate students and those new to qualitative research to identify and apply rigorous principles and indicators in qualitative research. The research reported in this article focuses on creating a scale designed to evaluate the psychometric properties of the TACT framework. This involves analyzing the stability of its dimensions and understanding its effectiveness as a tool for teaching and research.Results: The study's findings indicate that the TACT framework, when assessed through the newly developed scale, exhibits stable dimensions consistent with rigorous qualitative research principles. The framework effectively guides postgraduate students and new researchers in assessing the rigor of qualitative research processes and outcomes.Discussion:The application of the TACT framework and its evaluation scale reveals several insights. Firstly, it demonstrates the framework's utility in bridging the gap in pedagogical tools for teaching rigor in qualitative research methods. Secondly, it highlights the framework's potential in providing a structured approach to undertaking qualitative research, which is essential given this field's diverse methodologies and paradigms. However, the TACT framework remains a guide to enhancing rigor in qualitative research throughout all the various phases but by no means a measure of rigor.Conclusion:In conclusion, the TACT framework and its accompanying evaluative scale represent significant steps toward standardizing and enhancing the rigor of qualitative research, particularly for postgraduate students and early career researchers. While it does not solve all challenges associated with obtaining and demonstrating rigor in qualitative research, it provides a valuable tool for assessing and ensuring research quality, thereby addressing some of the longstanding criticisms of the quality of research obtained through qualitative methods

Quality of Information in Mobile Crowdsensing: Survey and Research Challenges

Smartphones have become the most pervasive devices in people's lives, and are clearly transforming the way we live and perceive technology. Today's smartphones benefit from almost ubiquitous Internet connectivity and come equipped with a plethora of inexpensive yet powerful embedded sensors, such as accelerometer, gyroscope, microphone, and camera. This unique combination has enabled revolutionary applications based on the mobile crowdsensing paradigm, such as real-time road traffic monitoring, air and noise pollution, crime control, and wildlife monitoring, just to name a few. Differently from prior sensing paradigms, humans are now the primary actors of the sensing process, since they become fundamental in retrieving reliable and up-to-date information about the event being monitored. As humans may behave unreliably or maliciously, assessing and guaranteeing Quality of Information (QoI) becomes more important than ever. In this paper, we provide a new framework for defining and enforcing the QoI in mobile crowdsensing, and analyze in depth the current state-of-the-art on the topic. We also outline novel research challenges, along with possible directions of future work.Comment: To appear in ACM Transactions on Sensor Networks (TOSN

Authenticity in Constructivist Inquiry: Assessing an Elusive Construct

Methodological rigor in constructivist inquiry is established through an assessment of trustworthiness and authenticity. Trustworthiness parallels the positivistic concepts of internal and external validity, focusing on an assessment of the inquiry process. Authenticity, however, is unique to constructivist inquiry and has no parallel in the positivistic paradigm. Authenticity involves an assessment of the meaningfulness and usefulness of interactive inquiry processes and social change that results from these processes. However, the techniques for ascertaining authenticity are in the early stages of development. Therefore, the purpose of this article is to describe a process for assessing authenticity in a constructivist inquiry. A brief overview of constructivist inquiries are described in relation to a set of techniques designed specifically to assess five dimensions of authenticity. Implications for constructivist researchers and social work research are presented