Cost and benefit analysis of quality requirements in competitive software product management: a case study on the QUPER model

In market-driven product development, it is important that the software product is released to the market at the right time, and offers higher quality than the competitors. In release planning, the allocation of development effort in investments into product enhancements, functions are competing with quality requirements for limited resources. In addition, it is important to find the right balance between competing quality requirements. In this paper, we present an industrial evaluation of the benefit and cost views in the QUality PERformance (QUPER) model. The results indicate that the model is relevant in the release planning process, and that the combination of the benefit and cost views provides a clear picture of the current market situation

A Case Study Evaluation of the Guideline-Supported QUPER Model for Elicitation of Quality Requirements

[Context & motivation] For market-driven software product developing organizations operating on a competitive open market, it is important to plan the product’s releases so that they can reach the market as early as possible with a competitive level of quality compared to its competitors' products. Hence, quality requirements can be seen as a key competitive advantage. The QUPER model was developed with the aim to support high-level decision-making in release planning of quality requirements. [Question/problem] As a follow up on previous studies on QUPER, this study investigates: What are practitioner's views on the utilities of QUPER extended with guidelines including domain-specific examples? [Principal ideas/results] In the presented case study, a set of detailed guidelines of how to apply QUPER in practice, including how to handle cost dependencies between quality requirements, was evaluated at a case company in the mobile handset domain with 24 professionals using real quality requirements. [Contribution] The results point to the importance of having concrete guidelines combined with instructive examples from real practice, while it is not always obvious for a practitioner to transfer cost-dependency examples into the domains that are different from the example domain. The transferability of guidelines and examples to support methodology adoption is an interesting issue for further research

Maps of Lessons Learnt in Requirements Engineering

Both researchers and practitioners have emphasized the importance of learning from past experiences and its consequential impact on project time, cost, and quality. However, from the survey we conducted of requirements engineering (RE) practitioners, over 70\% of the respondents stated that they seldom use RE lessons in the RE process, though 85\% of these would use such lessons if readily available. Our observation, however, is that RE lessons are scattered, mainly implicitly, in the literature and practice, which obviously, does not help the situation. We, therefore, present ``maps” of RE lessons which would highlight weak (dark) and strong (bright) areas of RE (and hence RE theories). Such maps would thus be: (a) a driver for research to ``light up” the darker areas of RE and (b) a guide for practice to benefit from the brighter areas. To achieve this goal, we populated the maps with over 200 RE lessons elicited from literature and practice using a systematic literature review and survey. The results show that approximately 80\% of the elicited lessons are implicit and that approximately 70\% of the lessons deal with the elicitation, analysis, and specification RE phases only. The RE Lesson Maps, elicited lessons, and the results from populating the maps provide novel scientific groundings for lessons learnt in RE as this topic has not yet been systematically studied in the field

Adopting Automated Bug Assignment in Practice: A Longitudinal Case Study at Ericsson

The continuous inflow of bug reports is a considerable challenge in large development projects. Inspired by contemporary work on mining software repositories, we designed a prototype bug assignment solution based on machine learning in 2011-2016. The prototype evolved into an internal Ericsson product, TRR, in 2017-2018. TRR's first bug assignment without human intervention happened in April 2019. Our study evaluates the adoption of TRR within its industrial context at Ericsson. Moreover, we investigate 1) how TRR performs in the field, 2) what value TRR provides to Ericsson, and 3) how TRR has influenced the ways of working. We conduct an industrial case study combining interviews with TRR stakeholders, minutes from sprint planning meetings, and bug tracking data. The data analysis includes thematic analysis, descriptive statistics, and Bayesian causal analysis. TRR is now an incorporated part of the bug assignment process. Considering the abstraction levels of the telecommunications stack, high-level modules are more positive while low-level modules experienced some drawbacks. On average, TRR automatically assigns 30% of the incoming bug reports with an accuracy of 75%. Auto-routed TRs are resolved around 21% faster within Ericsson, and TRR has saved highly seasoned engineers many hours of work. Indirect effects of adopting TRR include process improvements, process awareness, increased communication, and higher job satisfaction. TRR has saved time at Ericsson, but the adoption of automated bug assignment was more intricate compared to similar endeavors reported from other companies. We primarily attribute the difference to the very large size of the organization and the complex products. Key facilitators in the successful adoption include a gradual introduction, product champions, and careful stakeholder analysis.Comment: Under revie

From Bugs to Decision Support – Leveraging Historical Issue Reports in Software Evolution

Software developers in large projects work in complex information landscapes and staying on top of all relevant software artifacts is an acknowledged challenge. As software systems often evolve over many years, a large number of issue reports is typically managed during the lifetime of a system, representing the units of work needed for its improvement, e.g., defects to fix, requested features, or missing documentation. Efficient management of incoming issue reports requires the successful navigation of the information landscape of a project. In this thesis, we address two tasks involved in issue management: Issue Assignment (IA) and Change Impact Analysis (CIA). IA is the early task of allocating an issue report to a development team, and CIA is the subsequent activity of identifying how source code changes affect the existing software artifacts. While IA is fundamental in all large software projects, CIA is particularly important to safety-critical development. Our solution approach, grounded on surveys of industry practice as well as scientific literature, is to support navigation by combining information retrieval and machine learning into Recommendation Systems for Software Engineering (RSSE). While the sheer number of incoming issue reports might challenge the overview of a human developer, our techniques instead benefit from the availability of ever-growing training data. We leverage the volume of issue reports to develop accurate decision support for software evolution. We evaluate our proposals both by deploying an RSSE in two development teams, and by simulation scenarios, i.e., we assess the correctness of the RSSEs' output when replaying the historical inflow of issue reports. In total, more than 60,000 historical issue reports are involved in our studies, originating from the evolution of five proprietary systems for two companies. Our results show that RSSEs for both IA and CIA can help developers navigate large software projects, in terms of locating development teams and software artifacts. Finally, we discuss how to support the transfer of our results to industry, focusing on addressing the context dependency of our tool support by systematically tuning parameters to a specific operational setting

A Method for Evidence Based Quality Practice Engineering

Ohjelmistotuotannon laatu on yhä yksi IT-teollisuuden suurimpia ongelmia. Erityisen kiinnostava kysymys on, kuinka tietty laatutaso voitaisiin saavuttaa systemaattisesti. Empiirinen ohjelmistotuotanto (EBSE) pyrkii vastaamaan tähän kysymykseen keräämällä todistusaineistoa aitojen ohjelmistotuotantoprosessien ja -tuotoksien toimivuudesta. Tässä työssä tutkimusnäkökulmaksi on valittu laatukäytäntöjen ja -tavoitteiden välinen suhde konstruoimalla ja vertailemalla uusia menetelmä ohjelmistotuotantoprosessien kehittämiseksi. Tutkimusta varten suoritettiin yhteensä viisi toimintotutkimuksellista ja konstruktiivista interventiota neljässä suomalaisessa keskikokoisessa ohjelmistotuote-yrityksessä. Ensiksi sovellettiin laatupalettianalyysi -menetelmää kolmessa yrityksessä. Tämän jälkeen konsturoitiin uudet indikaattorianalyysi ja "NMA"-aivomyrsky menetelmät. Viimeisenä konstruktiona rakennettiin semanttisen verkon -teknologiaan pohjautuva tietämyskanta laatutavoitteiden ja -käytäntöjen välisten vaikutusten tutkimiseen. Tutkimuksen tulokset ovat kaksiosaiset. Huolimatta siitä, että todistusaineiston määrällisen puutteen takia tilastollisesti merkittäviä tuloksia ei voida tässä työssä esittää, uusi tietämyskanta vaikuttaisi teoreettisesti pystyvän tutkituista menetelmistä ainoana vastaamaan tutkimuskysymyksessä esitettyyn kysymykseen. Teollisuuden lisäksi tietämyskantaa on mahdollista hyödyntää akateemisessa tutkimuksessa, opetuksessa, koulutuksessa ja ohjelmistotuotantoprosessin kehitysideoiden arvioinnissa. Tietämyskanta pystyi vastaamaan tiettyihin kysymyksiin, kuten "mitä käytäntöjä tulisi soveltaa varmistuakseen, ettei ohjelmistopäivitykseen kuluva työmäärä ylitä 8h:a?". Vastauksena saatiin käytäntövektori "savutestiasennus" ja "alfa/beta -testaus". Vähäisen tutkimustiedon takia tietämyskanta ei kuitenkaan pysty toistaiseksi vastaamaan esimerkiksi miten päivitykseen kuluvaa työmäärää voitaisiin lyhentää. Lisäksi tuloksen luotettavuutta voidaan pitää heikkona. Tietokannan antamien vastauksien laajuutta ja luotettavuutta voitaisiin kuitenkin helposti parantaa laatimalla systemaattisia kirjallisuuskatsauksia kaikesta saatavilla olevasta ohjelmistotuotantokäytäntöihin liittyvästä tieteellisestä kirjallisuudesta ja syöttämällä tulokset tietämyskantaan. Kunnes tietämyskannasta pystytään kehittämään teolliseen käyttöön soveltuva versio, NMA -aivomyrsky on selkeästi tehokkain menetelmä kehitysideoiden tuottamiseen. Muut tutkitut työpajamenetelmät eivät aina tuottaneet lainkaan kehitysideoita eivätkä siten sovellu teolliseen käyttöön. Kuitenkin tämän työn lopussa esitetään ehdotuksia siitä, kuinka laatupalettianalyysiä voitaisiin hyödyntää tietämyskannan tiedonkeruumenetelmänä mm. poistamalla työpaja-vaihe ja kehittämällä uusi esitehtävätyökalu.The quality of the software has been and remains as a key problem of the software industry. Especially interesting questions is, how a certain level of quality can be systematically reached. Evidence-based software engineering (EBSE) tries to provide answer to this question by collecting empirical evidence on different aspects of the software engineering process and deliverables. In this work the perspective of quality practices and goals has been selected for constructing and evaluating a method that could be used for industrial software process improvement (SPI). Four subject Finnish middle-sized software product companies were studied by performing in total five action research and constructive interventions. First the novel Quality Palette Analysis -method was applied for three subject companies in different variations. Next the Indicator Analysis and the New Method A (NMA) -brainstorming method were constructed and applied by the author. As a final constructive step, the author designed a novel Semantic Web -based EBSE experience factory for mapping the empirical evidence on the relationship of the quality goals and practices. The results of the study are two-fold. While the collected data on the relationship between the quality goals and practices remains insufficient to draw definitive claims, it seems that the EBSE DB provides theoretically a very high utility model for software process improvement (SPI) initiative evaluation, training & education, and for the scientific research. The database is able to answer questions such as "which practices should be used to ensure reaching of effort of 8h per update" with an answer vector of practices "smokeTesting" and "alphaBetaTesting". Due to small amount of samples the database is currently unable to answer for example, how an update effort of 1 hour less could be reached and the results can be considered unreliable. However, the reliability and range of answerable questions could be easily improvement by performing systematic literature review on all available scientific evidence on the software engineering practices. While such a system remains as a prototype, the NMA -brainstorming method provided clearly the best yield of SPI initiatives compared to the time invested in the data collection. The other methods were by the best cumbersome and can't be recommended for industrial application in their current forms. However, the author provides suggestions how the QPA-method could be altered to function in the Future as a primary data collection method for the EBSE DB in context of individual companies by omitting the workshop -phase and developing an automatic data collection tool similar to the current QPA pre-assignment

Understanding and supporting large-scale requirements management

Large market-driven software companies face new challenges in requirements engineering and management that emerged due to their recent extensive growth. At the same time, the pressure generated by competitors’ and users’ expectations demands being more competitive, creative and flexible to more quickly respond to a rapidly changing market situation. In the pursuit of staying competitive in this context, new ideas on how to improve the current software engineering practice are requested to help maintaining the engineering efficiency while coping with growing size and complexity of requirements engineering processes and their products. This thesis focuses on understanding and supporting large-scale requirements management for developing software products to open markets. In particular, this thesis focuses on the following requirements management activities in the mentioned context, namely: scope management, variability management and requirements consolidation. The goals of the research effort in this thesis are to provide effective methods in supporting mentioned requirements management activities in a situation when the size of them and their complexity require large time and skills efforts. Based on empirical research, where both quantitative and qualitative approaches were utilized, this thesis reports on possible improvements for managing variability and presents visualization techniques to assist scope management for large-scale software product development contexts. Both reported ideas are empirically evaluated in case studies in a large-scale context. Additionally, the benefits of using linguistic methods for requirements consolidation are investigated in a replicated experimental study based on a relevant industry scenario

Linking Long-Term Product Planning and Requirements Prioritization to Customer Value Creation

Creating value for different customer segments is essential to the business of a company. Thus, software product development companies' ability to implement the most valuable requirements in their products has been seen as critical. The literature offers requirements prioritization methods for selecting requirements, but their suitability for solving practical challenges is not clear. The state of the practice in long-term product planning and requirements prioritization, and the practical challenges involved is not thoroughly analysed. Therefore, the connection between the selection of product features and customer value creation is also an area that needs more investigation. This thesis investigates the current state of long-term product planning and requirements prioritization, and their linkages to customer value creation in market-driven software product development. The results are based on the experience gathered from 7 Finnish software product development companies that had recognized the importance to improve their long-term product planning and requirements prioritization practices. To gain a deep understanding in real product development context, we conducted longitudinal case studies using an action research approach. In addition, we evaluated the suitability of requirements prioritization methods in product development projects. The thesis provides a systematic analysis of long-term product planning and requirements prioritization activities and challenges involved in the market-driven software product development companies. According to our results product planning activities are at a low-level, focusing on the individual features of individual products in short term. Practitioners have challenges in drawing different viewpoints together into product planning. Both integrating business and customer viewpoints to the project level as well as integrating the understanding about technical limitations and possibilities to product level planning was difficult in practice. The requirements prioritization methods did not solve the practical challenges. On the basis of our findings we also propose a set of practices that support the link from long-term product planning and requirements prioritization to customer value creation. The essence of the suggested practices is 1) in the shift of planning focus from individual product features towards understanding the processes of customers and users and in describing the solutions from the customer viewpoint as a whole and 2) in the cross-functional effort to be able to combine the three main viewpoints of customer value creation (company's own business, customers and users, and implementation).Arvon luominen eri asiakassegmenteille on elintärkeää yrityksen liiketoiminnalle. Ohjelmistotuoteyrityksen kyvykkyys toteuttaa tuotteisiinsa asiakkaiden näkökulmasta katsoen arvokkaimmat ominaisuudet onkin siksi nähty kriittisenä. Kirjallisuudessa esitellään erilaisia priorisointimenetelmiä tuoteominaisuuksien valintaan, mutta niiden soveltuvuudesta käytännön ohjelmistotyöhön ei juuri tiedetä. Myöskään pitkäntähtäimen suunnittelun ja vaatimusten priorisoinnin nykytilaa sekä niihin liittyviä käytännön haasteita ei ole syvällisesti analysoitu. Siksi yhteyttä ominaisuusvalintojen ja asiakasarvon luomisen välillä onkin tärkeää tutkia lisää. Tässä tutkimuksessa selvitettiin, miten pitkäntähtäimen tuotesuunnittelu ja vaatimusten priorisointi tukevat arvon luontia asiakkaille ja käyttäjille markkinaohjautuvassa ohjelmistotuotekehityksessä. Tulokset perustuvat seitsemästä suomalaisesta ohjelmistotuoteyrityksestä kerättyihin kokemuksiin. Saadaksemme tuotesuunnittelusta ja ominaisuusvalintojen tekemisestä niin pitkällä kuin lyhyelläkin tähtäimellä syvällisen ymmärryksen, toteutimme pitkittäisiä tapaustutkimuksia käyttäen toimintatutkimuslähestymistapaa. Lisäksi kokeilimme vaatimusten priorisointimenetelmiä todellisessa tuotekehitysympäristössä. Tutkimus esittelee systemaattisen nykytila-analyysin ohjelmistoyritysten tuotesuunnittelun käytännöistä ja haasteista. Tulokset paljastavat, että ohjelmistotuotteita suunnitellaan pirstaleisesti, lyhyellä aikajänteellä ja yksittäisen tuotteen ominaisuuksiin kerralla keskittyen. Yritysten työntekijöillä on vaikeuksia käytännössä yhdistää arvonluonnin kannalta tärkeitä näkökulmia. Sekä liiketoiminta- ja asiakasnäkökulman tuominen tuotekehitysprojektien päätöksentekoon että toisaalta teknisten rajoitteiden ja mahdollisuuksien huomioiminen jo tuotesuunnittelun aiemmissa vaiheissa ovat erityisen haastavia. Tutkimuksen tuloksena syntyi käytäntöjä, joiden avulla asiakasarvon luomista voisi tukea tuotesuunnittelussa. Näiden käytäntöjen keskiössä ovat 1) suunnittelun fokuksen siirtäminen yksittäisistä tuoteominaisuuksista asiakkaan ja loppukäyttäjän prosessien ymmärtämiseen, sekä ratkaisujen kuvaaminen asiakkaan näkökulmasta kokonaisuuksina, sekä 2) yli organisaatiorajojen ulottuva työskentely, jonka avulla voidaan paremmin yhdistää kolme tärkeintä asiakasarvon luonnin näkökulmaa (yrityksen oma liiketoiminta, asiakkaat ja käyttäjät, sekä tuotteen toteuttaminen)