Patterns of User Involvement in Experiment-Driven Software Development

Background: Experiments are often used as a means to continuously validate user needs and to aid in making software development decisions. Involving users in the development of software products benefits both the users and companies. How software companies efficiently involve users in both general development and in experiments remains unclear; however, it is especially important to determine the perceptions and attitudes held by practitioners in different roles in these companies. Objective: We seek to: 1) explore how software companies involve users in software development and experimentation; 2) understand how developer, manager and UX designer roles perceive and involve users in experimentation; and 3) uncover systematic patterns in practitioners’ views on user involvement in experimentation. The study aims to reveal behaviors and perceptions that could support or undermine experiment-driven development, point out what skills could enhance experiment-driven development, and raise awareness of such issues for companies that wish to adopt experiment-driven development. Methods: We conducted a survey within four Nordic software companies, inviting practitioners in three major roles: developers, managers, and UX designers. We asked the respondents to indicate how they involve users in their job function, as well as their perspectives regarding software experiments and ethics. Results and Conclusion: We identified six patterns describing experimentation and user involvement. For instance, managers were associated with a cautious user notification policy, that is, to always let users know of an experiment they are subject to, and they also believe that users have to be convinced before taking part in experiments. We discovered that, due to lack of clear processes for involving users and the lack of a common understanding of ethics in experimentation, practitioners tend to rationalize their perceptions based on their own experiences. Our patterns were based on empirical evidence and they can be evaluated in different populations and contexts.Peer reviewe

Yhteisen tietopohjan ylläpidon välineet ketterissä ohjelmistoprojekteissa

Ketterä ohjelmistokehitys syntyi vastauksena yritysmaailman nopeasti muuttuviin tarpeisiin. Ketterät menetelmät painottavat tehokasta epämuodollista viestintää ja yhteistyötä asiakkaan kanssa. Tutkielmassa tarkastellaan, millä välineillä ketterissä projekteissa ylläpidetään asiakkaan ja kehitystiimin välillä yhteistä tietopohjaa, eli yhteisesti jaettua käsitystä tarpeista ja projektin tilanteesta, ja miten ne soveltuvat tarkoitukseensa. Tutkielma on usean tapauksen tapaustutkimus, jonka aineisto on kerätty kolmesta ohjelmistoyrityksestä puolistrukturoiduilla haastatteluilla. Haastatteluihin saatiin rekrytoitua kuusi ohjelmistoprojekteissa eri tehtävissä toiminutta henkilöä. Heitä pyydettiin miettimään yhtä esimerkkiprojektia haastattelua varten. Haastateltujen esittelemät projektit olivat hyvin erilaisia jopa saman yrityksen sisällä, joten niitä oli mielekästä tarkastella omina tapauksinaan. Aineiston analyysivaiheessa haastatteluissa esiin nousseiden viestintämedioiden soveltuvuutta viestintätarpeisiin arvioitiin median synkronisaation teorian pohjalta. Viestinnällä oli suuri rooli jokaisessa projektissa ja yhdessäkään projektissa ei tyydytty käyttämään vain yhtä viestintämediaa. Haastateltujen esittelemissä projekteissa käytettyjen viestintämedioiden käyttötavat noudattelivat pääasiassa median synkronisaation teorian esittämiä suosituksia siitä, millaisiin viestintätarpeisiin mitkäkin viestintämediat sopivat. Muutamia poikkeuksia näihin suosituksiin ilmeni. Esimerkiksi kiireellisissä tilanteissa käytettiin korkean synkronisaation viestintämedioita viestintätarpeisiin, joihin matalan synkronisaation mediat olisivat teorian mukaan sopineet paremmin. Aikataulupaine ajoi muiden tarpeiden edelle ja vastausten saamista nopeasti pidettiin tärkeämpänä kuin viestin parempaa ymmärrettävyyttä

On Experimentation in Software-Intensive Systems

Context: Delivering software that has value to customers is a primary concern of every software company. Prevalent in web-facing companies, controlled experiments are used to validate and deliver value in incremental deployments. At the same that web-facing companies are aiming to automate and reduce the cost of each experiment iteration, embedded systems companies are starting to adopt experimentation practices and leverage their activities on the automation developments made in the online domain. Objective: This thesis has two main objectives. The first objective is to analyze how software companies can run and optimize their systems through automated experiments. This objective is investigated from the perspectives of the software architecture, the algorithms for the experiment execution and the experimentation process. The second objective is to analyze how non web-facing companies can adopt experimentation as part of their development process to validate and deliver value to their customers continuously. This objective is investigated from the perspectives of the software development process and focuses on the experimentation aspects that are distinct from web-facing companies. Method: To achieve these objectives, we conducted research in close collaboration with industry and used a combination of different empirical research methods: case studies, literature reviews, simulations, and empirical evaluations. Results: This thesis provides six main results. First, it proposes an architecture framework for automated experimentation that can be used with different types of experimental designs in both embedded systems and web-facing systems. Second, it proposes a new experimentation process to capture the details of a trustworthy experimentation process that can be used as the basis for an automated experimentation process. Third, it identifies the restrictions and pitfalls of different multi-armed bandit algorithms for automating experiments in industry. This thesis also proposes a set of guidelines to help practitioners select a technique that minimizes the occurrence of these pitfalls. Fourth, it proposes statistical models to analyze optimization algorithms that can be used in automated experimentation. Fifth, it identifies the key challenges faced by embedded systems companies when adopting controlled experimentation, and we propose a set of strategies to address these challenges. Sixth, it identifies experimentation techniques and proposes a new continuous experimentation model for mission-critical and business-to-business. Conclusion: The results presented in this thesis indicate that the trustworthiness in the experimentation process and the selection of algorithms still need to be addressed before automated experimentation can be used at scale in industry. The embedded systems industry faces challenges in adopting experimentation as part of its development process. In part, this is due to the low number of users and devices that can be used in experiments and the diversity of the required experimental designs for each new situation. This limitation increases both the complexity of the experimentation process and the number of techniques used to address this constraint

Towards Continuous Validation of Customer Value

While close customer collaboration is highlighted as a distinguishing characteristic in agile development, difficulties arise in large-scale agile development where the product owner can no longer represent the different needs of a large customer base. While most companies use the role of a product owner to represent the customer base, experiences show that prioritizations that are made are far from optimal. Also, once the decision to develop a feature has been taken, companies stop to continuously validate if this feature adds value to the large customer base. As experienced in the case companies we work with, re-prioritization of feature content is difficult once development has started, resulting in R&D investments in development of features that have no proven customer value. In this paper, and based on our experiences from working with five B2B software development companies, we present a conceptual model in which qualitative and quantitative customer feedback techniques allow for continuous validation and re-prioritization of feature content. In this way, large-scale software development companies can significantly improve responsiveness to customers throughout the development cycle, while at the same time increase accuracy of their development efforts

Towards continuous validation of customer value

While close customer collaboration is highlighted as a distinguishing characteristic in agile development, difficulties arise in large-scale agile development where the product owner can no longer represent the different needs of a large customer base. While most companies use the role of a product owner to represent the customer base, experiences show that prioritizations that are made are far from optimal. Also, once the decision to develop a feature has been taken, companies stop to continuously validate if this feature adds value to the large customer base. As experienced in the case companies we work with, re-prioritization of feature content is difficult once development has started, resulting in R&D investments in development of features that have no proven customer value. In this paper, and based on our experiences from working with five B2B software development companies, we present a conceptual model in which qualitative and quantitative customer feedback techniques allow for continuous validation and re-prioritization of feature content. In this way, large-scale software development companies can significantly improve responsiveness to customers throughout the development cycle, while at the same time increase accuracy of their development efforts