Laboratory experiments as evaluating method for IT artifacts – Experimental design and use case within a Product-Service-System

According to the paradigm of Design Science Research, IT artifacts are developed and built to solve real-life problems that occur in business life. Since Design Science Research consists of two phases, it is divided in the building and the evaluation phase. The evaluation phase is considered to be highly important as it analyses the relevance and the functionality of the artifact. To evaluate if an artifact meets the users’ needs, experiments are a possible method. Based on findings in a current research project, we present an experiment design to evaluate artifacts being part of Product-Service-Systems. The design is applied in a use case and supplemented by statistical data to analyze how the artifact meets the users’ needs

The Usage and Evaluation of Anthropomorphic Form in Robot Design

There are numerous examples illustrating the application of human shape in everyday products. Usage of anthropomorphic form has long been a basic design strategy, particularly in the design of intelligent service robots. As such, it is desirable to use anthropomorphic form not only in aesthetic design but also in interaction design. Proceeding from how anthropomorphism in various domains has taken effect on human perception, we assumed that anthropomorphic form used in appearance and interaction design of robots enriches the explanation of its function and creates familiarity with robots. From many cases we have found, misused anthropomorphic form lead to user disappointment or negative impressions on the robot. In order to effectively use anthropomorphic form, it is necessary to measure the similarity of an artifact to the human form (humanness), and then evaluate whether the usage of anthropomorphic form fits the artifact. The goal of this study is to propose a general evaluation framework of anthropomorphic form for robot design. We suggest three major steps for framing the evaluation: 'measuring anthropomorphic form in appearance', 'measuring anthropomorphic form in Human-Robot Interaction', and 'evaluation of accordance of two former measurements'. This evaluation process will endow a robot an amount of humanness in their appearance equivalent to an amount of humanness in interaction ability, and then ultimately facilitate user satisfaction. Keywords: Anthropomorphic Form; Anthropomorphism; Human-Robot Interaction; Humanness; Robot Design</p

Phantoms of Innovation: Disciplined Simulation for Ex-ante Evaluation in Design Science Research

For years, there has been an emphasis on how to efficiently and effectively identify, evaluate, and implement innovative information systems in both design science research (DSR) and practice. Nonetheless, still today, these efforts continue to be hampered by the temporal gap between ideation and evaluation. Usually, innovative ideas are implemented at a late stage of maturity (e.g., prototypes) to test their viability in practice. This widespread approach results in waste of resources and time if the viability of an idea fails outside the lab environment. This paper discusses an ex-ante evaluation approach derived from “pretotyping” that allows innovative ideas to be tested in naturalistic settings even before they have been implemented. Thus, we call them “phantoms”. We show how this approach reduces temporal and relevance gaps, and we provide a preliminary assessment of its practicability by presenting and discussing three case studies conducted with real organizations and prospective users

Sensemaking Practices in the Everyday Work of AI/ML Software Engineering

This paper considers sensemaking as it relates to everyday software engineering (SE) work practices and draws on a multi-year ethnographic study of SE projects at a large, global technology company building digital services infused with artificial intelligence (AI) and machine learning (ML) capabilities. Our findings highlight the breadth of sensemaking practices in AI/ML projects, noting developers' efforts to make sense of AI/ML environments (e.g., algorithms/methods and libraries), of AI/ML model ecosystems (e.g., pre-trained models and "upstream"models), and of business-AI relations (e.g., how the AI/ML service relates to the domain context and business problem at hand). This paper builds on recent scholarship drawing attention to the integral role of sensemaking in everyday SE practices by empirically investigating how and in what ways AI/ML projects present software teams with emergent sensemaking requirements and opportunities

On the Use of Experiments in Design Science Research: A Proposition of an Evaluation Framework

Although experiments are considered a valid scientific method for evaluating the outcome of design science research (DSR), only minimal procedural guidelines or standards exist that help researchers in the setup and conduct. To take advantage of and encourage researchers to include design experiments as an inherent part of their build and test cycle, this study proposes a set of guidelines. In order to get a broad overview of how researchers currently apply the experimental method in DSR and to detect potential drawbacks, an extensive review of the extant literature was conducted. On this basis, we propose an evaluation framework that complements the general design science research guidelines of Hevner and colleagues. The purpose of this framework is to assist researchers, reviewers, editors, and readers in understanding possible pitfalls as well as to ask the right questions which need to be answered in the conduct of design experiments

Flexible Global Software Development (GSD): Antecedents of Success in Requirements Analysis

Globalization of software development has resulted in a rapid shift away from the traditional collocated, on-site development model, to the offshoring model. Emerging trends indicate an increasing interest in offshoring even in early phases like requirements analysis. Additionally, the flexibility offered by the agile development approach makes it attractive for adaptation in globally distributed software work. A question of significance then is what impacts the success of offshoring earlier phases, like requirements analysis, in a flexible and globally distributed environment? This article incorporates the stance of control theory to posit a research model that examines antecedent factors such as requirements change, facilitation by vendor and client site-coordinators, control, and computer-mediated communication. The impact of these factors on success of requirements analysis projects in a “flexible” global setting is tested using two quasi-experiments involving students from Management Development Institute, India and Marquette University, USA. Results indicate that formal modes of control significantly influence project success during requirements analysis. Further, facilitation by both client and vendor site coordinators positively impacts requirements analysis success

A Usability Inspection Method for Model-driven Web Development Processes

Las aplicaciones Web son consideradas actualmente un elemento esencial e indispensable en toda actividad empresarial, intercambio de información y motor de redes sociales. La usabilidad, en este tipo de aplicaciones, es reconocida como uno de los factores clave más importantes, puesto que la facilidad o dificultad que los usuarios experimentan con estas aplicaciones determinan en gran medida su éxito o fracaso. Sin embargo, existen varias limitaciones en las propuestas actuales de evaluación de usabilidad Web, tales como: el concepto de usabilidad sólo se soporta parcialmente, las evaluaciones de usabilidad se realizan principalmente cuando la aplicación Web se ha desarrollado, hay una carencia de guías sobre cómo integrar adecuadamente la usabilidad en el desarrollo Web, y también existe una carencia de métodos de evaluación de la usabilidad Web que hayan sido validados empíricamente. Además, la mayoría de los procesos de desarrollo Web no aprovechan los artefactos producidos en las fases de diseño. Estos artefactos software intermedios se utilizan principalmente para guiar a los desarrolladores y para documentar la aplicación Web, pero no para realizar evaluaciones de usabilidad. Dado que la trazabilidad entre estos artefactos y la aplicación Web final no está bien definida, la realización de evaluaciones de usabilidad de estos artefactos resulta difícil. Este problema se mitiga en el desarrollo Web dirigido por modelos (DWDM), donde los artefactos intermedios (modelos) que representan diferentes perspectivas de una aplicación Web, se utilizan en todas las etapas del proceso de desarrollo, y el código fuente final se genera automáticamente a partir estos modelos. Al tener en cuenta la trazabilidad entre estos modelos, la evaluación de estos modelos permite detectar problemas de usabilidad que experimentaran los usuarios finales de la aplicación Web final, y proveer recomendaciones para corregir estos problemas de usabilidad durante fases tempranas del proceso de desarrollo Web. Esta tesis tiene como objetivo, tratando las anteriores limitaciones detectadas, el proponer un método de inspección de usabilidad que se puede integrar en diferentes procesos de desarrollo Web dirigido por modelos. El método se compone de un modelo de usabilidad Web que descompone el concepto de usabilidad en sub-características, atributos y métricas genéricas, y un proceso de evaluación de usabilidad Web (WUEP), que proporciona directrices sobre cómo el modelo de usabilidad se puede utilizar para llevar a cabo evaluaciones específicas. Las métricas genéricas del modelo de usabilidad deben operacionalizarse con el fin de ser aplicables a los artefactos software de diferentes métodos de desarrollo Web y en diferentes niveles de abstracción, lo que permite evaluar la usabilidad en varias etapas del proceso de desarrollo Web, especialmente en las etapas tempranas. Tanto el modelo de usabilidad como el proceso de evaluación están alineados con la última norma ISO/IEC 25000 estándar para la evaluación de la calidad de productos de software (SQuaRE). El método de inspección de usabilidad propuesto (WUEP) se ha instanciado en dos procesos de desarrollo Web dirigido por modelos diferentes (OO-H y WebML) a fin de demostrar la factibilidad de nuestra propuesta. Además, WUEP fue validado empíricamente mediante la realización de una familia de experimentos en OO-H y un experimento controlado en WebML. El objetivo de nuestros estudios empíricos fue evaluar la efectividad, la eficiencia, facilidad de uso percibida y la satisfacción percibida de los participantes; cuando utilizaron WUEP en comparación con un método de inspección industrial ampliamente utilizado: La Evaluación Heurística (HE). El análisis estadístico y meta-análisis de los datos obtenidos por separado de cada experimento indicaron que WUEP es más eficaz y eficiente que HE en la detección de problemas de usabilidad. Los evaluadores también percibieron más satisfacción cuando se aplicaron WUEP, y lesFernández Martínez, A. (2012). A Usability Inspection Method for Model-driven Web Development Processes [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/17845Palanci