Ciriello, Raffaele

Cito, Jürgen

Loss, Sebastian

Crossref

Beware of Disengaged User Acceptance in Testing Software-as-a-Service

User acceptance tests (UAT) are an integral part of software engineering. This study aims to question the appropriateness of UATs to collect usable feedback for Software-as-a-Service (SaaS) applications, which are continuously delivered rather than rolled out during a one-off signoff process. Our preliminary results from an exploratory qualitative field study at a multinational SaaS provider in Denmark show that UATs often address the wrong problem in that positive user acceptance may paradoxically indicate a negative user experience. Hence, SaaS providers should be careful not to rest on what we initially term disengaged user acceptance. Instead, we aim to explore how SaaS providers can purposefully query users for ambivalent emotions to evoke constructive criticism. We briefly outline the adverse effects of disengaged user acceptance on testing SaaS applications

The IT University of Copenhagen's Repository

Pre-Print / Author Version. To appear in the Proceedings of the 41st International Conference on Software Engineering (ICSE19), Montréal, Canada Beware of Disengaged User Acceptance in Testing Software-as-a-Service    Sebastian Loss  Business IT Department IT University of Copenhagen Copenhagen, Denmark selo@itu.dk Raffaele Fabio Ciriello Business IT Department  IT University of Copenhagen Copenhagen, Denmark raci@itu.dk Jürgen Cito MIT CSAIL  Massachusetts Institute of Technology Cambridge, MA jcito@mit.edu  Abstract— User acceptance tests (UAT) are an integral part of software engineering. This study aims to question the appropriateness of UATs to collect usable feedback for Software-as-a-Service (SaaS) applications, which are continuously delivered rather than rolled out during a one-off signoff process. Our preliminary results from an exploratory qualitative field study at a multinational SaaS provider in Denmark show that UATs often address the wrong problem in that positive user acceptance may paradoxically indicate a negative user experience. Hence, SaaS providers should be careful not to rest on what we initially term disengaged user acceptance. Instead, we aim to explore how SaaS providers can purposefully query users for ambivalent emotions to evoke constructive criticism. We briefly outline the adverse effects of disengaged user acceptance on testing SaaS applications. Keywords— user acceptance, testing, software-as-a-service, cloud computing, constructive criticism, continuous delivery  I. INTRODUCTION Software providers carry out user acceptance tests (UAT) to increase efficiency and quality of a new release to be rolled out [1] and to validate the product increment against identified acceptance criteria [2]. A user acceptance test (UAT) can be defined as a “formal testing conducted to enable a user, customer, or other authorized entity to determine whether to accept a product or product component” [3, p. 91]. UATs usually consist of a set of processes to validate product components against clearly defined acceptance criteria [4]. At first glance, conducting UATs seems reasonable, as users should have a voice in what will be released.  However, in contrast to custom-made, contractually ordered software products, Software-as-a-Service (SaaS) allows for live site usage monitoring and continuous delivery. Typically, SaaS providers set acceptance criteria while allowing users to experiment with a pre-release to discover problems and to provide further input [4]. Against this backdrop, it falls into question whether UATs are actually appropriate in SaaS. Could UATs be simply a vestige from plan-driven software development? Prior research has found no empirical evidence that a customer’s loyalty to the service provider correlates with how the customer rates the service on a satisfaction survey [5].  Hence, this study explores the human practices and effects of UATs in SaaS through a qualitative field study. We examine whether formal UAT practices of verifying a software build against a fixed set of acceptance criteria can yield valuable user feedback in a SaaS context. We ask:   What role do user acceptance tests play in Software-as-a-Service? II. RESEARCH APPROACH Our research approach is guided by the principles of interpretive field research, which aims to understand the deeper structure of a phenomenon from a participant’s perspective [6]. Figure 1 provides an overview of our research approach and the overall analytical process. To date, we obtained data from a multinational SaaS provider in Denmark with a substantial and diverse user base. Data collection proceeded between February and May 2018. Our primary data sources were 11 expert interviews, as these allow to access the different perceptions and interpretations of participants [7], complemented by observations and archival documents.  Relying on qualitative data analysis techniques, such as coding, our data analysis and interpretation was primarily data-driven while also comparing the emerging concepts with existing theories that appeared plausible along the way [8]. Based on the collected data we identified possible categories (i.e. candidates for dominant themes) and their properties (i.e. candidates for sub-themes) through open coding techniques. By means of axial coding techniques, through which we established connections between emerging themes, we could then construct a more comprehensive scheme of the present practices and motivations in relation to UATs.  III. PRELIMINARY RESULTS Four preliminary results emerge from our exploratory case study on the practices and effects of user acceptance testing in SaaS: 1. UATs often serve the dual purpose of a legal signoff and a usability test, in which testers put themselves in the shoes of the user, leading to imagined acceptance that does not necessarily equal actual user acceptance.  2. UATs often address the wrong problem when measuring user satisfaction, especially when they conflict with live site usage goals, such as engagement, interaction, and emotional attachment. Test environment goals can be characterized as outcome-oriented and are steered towards acceptance, whereas live site environment goals are more processual and focus on actual usage.  3. UATs can hinder constructive criticism, as they do not allow for emotional reactions and communicating negative feedback appropriately. Instead, users fall into disengaged user acceptance, meaning that they passively conform with UAT procedures while hiding their actual emotional connection to the tested service. 4. Cultural factors may contribute to the spreading of disengaged user acceptance, further hindering constructive criticism. We found that negative feedback was held back, sugar-coated, or diluted, despite the corporate culture being perceived as open and supportive. Holding back criticism turned out to be a major barrier to product innovation. Pre-Print / Author Version. To appear in the Proceedings of the 41st International Conference on Software Engineering (ICSE19), Montréal, Canada  Figure 1. Overview of our research approach and preliminary findings. These preliminary findings are also visually represented in Figure 1, in which we describe how and during which phases imagined acceptance arises in user acceptance testing and how disengagement then continues along during live site usage. This vicious circle of disengaged user acceptance shows how mostly superficial user feedback is being transmitted, bearing the risk of users gradually detaching themselves from the continuous design process of a SaaS system. Ultimately, this can hinder SaaS providers to make user-centered improvements to their products. IV. CONCLUSION AND OUTLOOK From these preliminary results, we conclude that the question to ask should not be whether or not a customer accepts the SaaS product, and neither do UATs provide a sufficient answer. Quite the contrary, successful UATs can be an early warning sign of disengagement and passive compliance, while criticism and complaints can actually indicate that the customer is engaged and values the product. Hence, user acceptance testing alone cannot provide an accurate picture of how the user actually feels about the tested product or feature. Even worse, being overly reliant on UATs may put service providers at risk of "checking the wrong boxes", substituting actual user experience with user acceptance while users fall into what we term disengaged user acceptance.  Disengaged user acceptance can be understood as passively conforming with UAT procedures while hiding one's actual emotional connection to the tested SaaS product. Most importantly, our data shows that this may also occur due to UATs being the "right method to answer the wrong question".  Once disengaged user acceptance emerges, it continues to spread throughout the live site usage. This can potentially lead to a vicious circle, during which users detach themselves ever further from the SaaS application and its provider, leading to ever less engagement and, eventually, churn.  Hence, we suggest that UATs should play only a minor role in evaluating the overall user experience, as they can only provide a brief snapshot of a static situation. Rather than relying solely on UATs, cultivating constructive criticism could be more helpful for the continuous evolution of SaaS applications.  As a next step, we plan to explore how SaaS providers can query users for unfiltered and mixed emotional responses to the product. From a theoretical and practical perspective, this further research may help SaaS providers to avoid disengaged user acceptance and instead move to constructive criticism. It could also be interesting to include the learnings gathered from this research in a multi case study following up on this exploratory study. REFERENCES [1] J. Humble and D. Farley, Continuous delivery: reliable software releases through build, test, and deployment automation. Addison-Wesley Boston, 2011. [2] ISO, “ISO 9241-210:2010(en), Ergonomics of human-system interaction — Part 210: Human-centred design for interactive systems,” 2010. [Online]. Available: https://www.iso.org/obp/ui/#iso:std:iso:9241:-210:ed-1:v1:en. [Accessed: 28-Aug-2018]. [3]     A. Khannur, Structured Software Testing: The Discipline of Discovering. New Delhi: Partridge, 2014. [4] I. Sommerville, Software engineering. Addison-wesley, 2007. [5] M. Dixon, N. Toman, and R. DeLisi, The effortless experience: conquering the new battleground for customer loyalty. Penguin, 2013. [6] W. J. Orlikowski and J. J. Baroudi, “Studying information technology in organizations: Research approaches and assumptions,” Inf. Syst. Res., vol. 2, no. 1, pp. 1–28, 1991. [7] H. K. Klein and M. D. Myers, “A set of principles for conducting and evaluating interpretive field studies in information systems,” MIS Q., pp. 67–93, 1999. [8] J. T. DeCuir-Gunby, P. L. Marshall, and A. W. McCulloch, “Developing and using a codebook for the analysis of interview data: An example from a professional development research project,” Field Methods, vol. 23, no. 2, pp. 136–155, 2011.

https://pure.itu.dk/ws/files/84115583/ICSE19_Disengaged_User_Acceptance_Cam_Ready.pdf

Beware of Disengaged User Acceptance in Testing Software-as-a-Service

Abstract

Similar works

Full text

Available Versions

Crossref

The IT University of Copenhagen's Repository