A systematic review of quality attributes and measures for software product lines

[EN] It is widely accepted that software measures provide an appropriate mechanism for understanding, monitoring, controlling, and predicting the quality of software development projects. In software product lines (SPL), quality is even more important than in a single software product since, owing to systematic reuse, a fault or an inadequate design decision could be propagated to several products in the family. Over the last few years, a great number of quality attributes and measures for assessing the quality of SPL have been reported in literature. However, no studies summarizing the current knowledge about them exist. This paper presents a systematic literature review with the objective of identifying and interpreting all the available studies from 1996 to 2010 that present quality attributes and/or measures for SPL. These attributes and measures have been classified using a set of criteria that includes the life cycle phase in which the measures are applied; the corresponding quality characteristics; their support for specific SPL characteristics (e. g., variability, compositionality); the procedure used to validate the measures, etc. We found 165 measures related to 97 different quality attributes. The results of the review indicated that 92% of the measures evaluate attributes that are related to maintainability. In addition, 67% of the measures are used during the design phase of Domain Engineering, and 56% are applied to evaluate the product line architecture. However, only 25% of them have been empirically validated. In conclusion, the results provide a global vision of the state of the research within this area in order to help researchers in detecting weaknesses, directing research efforts, and identifying new research lines. In particular, there is a need for new measures with which to evaluate both the quality of the artifacts produced during the entire SPL life cycle and other quality characteristics. There is also a need for more validation (both theoretical and empirical) of existing measures. In addition, our results may be useful as a reference guide for practitioners to assist them in the selection or the adaptation of existing measures for evaluating their software product lines. © 2011 Springer Science+Business Media, LLC.This research has been funded by the Spanish Ministry of Science and Innovation under the MULTIPLE (Multimodeling Approach For Quality-Aware Software Product Lines) project with ref. TIN2009-13838.Montagud Gregori, S.; Abrahao Gonzales, SM.; Insfrán Pelozo, CE. (2012). A systematic review of quality attributes and measures for software product lines. Software Quality Journal. 20(3-4):425-486. https://doi.org/10.1007/s11219-011-9146-7S425486203-4Abdelmoez, W., Nassar, D. M., Shereschevsky, M., Gradetsky, N., Gunnalan, R., Ammar, H. H., et al. (2004). Error propagation in software architectures. In 10th international symposium on software metrics (METRICS), Chicago, Illinois, USA.Ajila, S. A., & Dumitrescu, R. T. (2007). Experimental use of code delta, code churn, and rate of change to understand software product line evolution. Journal of Systems and Software, 80, 74–91.Aldekoa, G., Trujillo, S., Sagardui, G., & Díaz, O. (2006). Experience measuring maintainability in software product lines. In XV Jornadas de Ingeniería del Software y Bases de Datos (JISBD). Barcelona.Aldekoa, G., Trujillo, S., Sagardui, G., & Díaz, O. (2008). Quantifying maintanibility in feature oriented product lines, Athens, Greece, pp. 243–247.Alves de Oliveira Junior, E., Gimenes, I. M. S., & Maldonado, J. C. (2008). A metric suite to support software product line architecture evaluation. In XXXIV Conferencia Latinamericana de Informática (CLEI), Santa Fé, Argentina, pp. 489–498.Alves, V., Niu, N., Alves, C., & Valença, G. (2010). Requirements engineering for software product lines: A systematic literature review. Information & Software Technology, 52(8), 806–820.Bosch, J. (2000). Design and use of software architectures: Adopting and evolving a product line approach. USA: ACM Press/Addison-Wesley Publishing Co.Briand, L. C., Differing, C. M., & Rombach, D. (1996a). Practical guidelines for measurement-based process improvement. Software Process-Improvement and Practice, 2, 253–280.Briand, L. C., Morasca, S., & Basili, V. R. (1996b). Property based software engineering measurement. IEEE Transactions on Software Eng., 22(1), 68–86.Calero, C., Ruiz, J., & Piattini, M. (2005). Classifying web metrics using the web quality model. Online Information Review, 29(3): 227–248.Chen, L., Ali Babar, M., & Ali, N. (2009). Variability management in software product lines: A systematic review. In 13th international software product lines conferences (SPLC), San Francisco, USA.Clements, P., & Northrop, L. (2002). Software product lines. 2003. Software product lines practices and patterns. Boston, MA: Addison-Wesley.Crnkovic, I., & Larsson, M. (2004). Classification of quality attributes for predictability in component-based systems. Journal of Econometrics, pp. 231–250.Conference Rankings of Computing Research and Education Association of Australasia (CORE). (2010). Available in http://core.edu.au/index.php/categories/conference%20rankings/1 .Davis, A., Dieste, Ó., Hickey, A., Juristo, N., & Moreno, A. M. (2006). Effectiveness of requirements elicitation techniques: Empirical results derived from a systematic review. In 14th IEEE international conference requirements engineering, pp. 179–188.de Souza Filho, E. D., de Oliveira Cavalcanti, R., Neiva, D. F. S., Oliveira, T. H. B., Barachisio Lisboa, L., de Almeida E. S., & de Lemos Meira, S. R. (2008). Evaluating domain design approaches using systematic review. In 2nd European conference on software architecture, Cyprus, pp. 50–65.Ejiogu, L. (1991). Software engineering with formal metrics. QED Publishing.Engström, E., & Runeson, P. (2011). Software product line testing—A systematic mapping study. Information & Software Technology, 53(1), 2–13.Etxeberria, L., Sagarui, G., & Belategi, L. (2008). Quality aware software product line engineering. Journal of the Brazilian Computer Society, 14(1), Campinas Mar.Ganesan, D., Knodel, J., Kolb, R., Haury, U., & Meier, G. (2007). Comparing costs and benefits of different test strategies for a software product line: A study from Testo AG. In 11th international software product line conference, Kyoto, Japan, pp. 74–83, September 2007.Gómez, O., Oktaba, H., Piattini, M., & García, F. (2006). A systematic review measurement in software engineering: State-of-the-art in measures. In First international conference on software and data technologies (ICSOFT), Setúbal, Portugal, pp. 11–14.IEEE standard for a software quality metrics methodology, IEEE Std 1061-1998, 1998.Inoki, M., & Fukazawa, Y. (2007). Software product line evolution method based on Kaizen approach. In 22nd annual ACM symposium on applied computing, Korea.Insfran, E., & Fernandez, A. (2008). A systematic review of usability evaluation in Web development. 2nd international workshop on web usability and accessibility (IWWUA’08), New Zealand, LNCS 5176, Springer, pp. 81–91.ISO/IEC 25010. (2008). Systems and software engineering. Systems and software Quality Requirements and Evaluation (SQuaRE). System and software quality models.ISO/IEC 9126. (2000). Software engineering. Product Quality.Johansson, E., & Höst, R. (2002). Tracking degradation in software product lines through measurement of design rule violations. In 14th International conference on software engineering and knowledge engineering, Ischia, Italy, pp. 249–254.Journal Citation Reports of Thomson Reuters. (2010). Available in http://thomsonreuters.com/products_services/science/science_products/a-z/journal_citation_reports/ .Khurum, M., & Gorschek, T. (2009). A systematic review of domain analysis solutions for product lines. The Journal of Systems and Software.Kim, T., Ko, I. Y., Kang, S. W., & Lee, D. H. (2008). Extending ATAM to assess product line architecture. In 8th IEEE international conference on computer and information technology, pp. 790–797.Kitchenham, B. (2007). Guidelines for performing systematic literature reviews in software engineering. Version 2.3, EBSE Technical Report, Keele University, UK.Kitchenham, B., Pfleeger, S., & Fenton, N. (1995). Towards a framework for software measurement validation. IEEE Transactions on Software Engineering, 21(12).Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.Mendes, E. (2005). A systematic review of Web engineering research. International symposium on empirical software engineering. Noosa Heads, Australia.Meyer, M. H., & Dalal, D. (2002). Managing platform architectures and manufacturing processes for non assembled products. Journal of Product Innovation Management, 19(4), 277–293.Montagud, S., & Abrahão, S. (2009). Gathering Current knowledge about quality evaluation in software product lines. In 13th international software product lines conferences (SPLC), San Francisco, USA.Montagud, S., & Abrahão, S. (2009). A SQuaRE-bassed quality evaluation method for software product lines. Master’s thesis, December 2009 (in Spanish).Needham, D., & Jones, S. (2006). A software fault tree metric. In 22nd international conference on software maintenance (ICSM), Philadelphia, Pennsylvania, USA.Niemelä, E., & Immonen, A. (2007). Capturing quality requirements of product family architecture. Information and Software Technology, 49(11–12), 1107–1120.Odia, O. E. (2007). Testing in software product lines. Master Thesis Software Engineering of School of Engineering, Bleking Institute of Technology. Thesis no. MSE-2007:16, Sweden.Olumofin, F. G., & Mišić, V. B. (2007). A holistic architecture assessment method for software product lines. Information and Software Technology, 49, 309–323.Pérez Lamancha, B., Polo Usaola, M., & Piattini Velthius, M. (2009). Software product line testing—a systematic review. ICSOFT, (1), 23–30.Poels, G., & Dedene, G. (2000). Distance-based software measurement: necessary and sufficient properties for software measures. Information and Software Technology, 42(I), 35–46.Prehofer, C., van Gurp, J., & Bosch, J. (2008). Compositionality in software platforms. In Emerging methods, technologies and process management in software engineering. Wiley.Rahman, A. (2004). Metrics for the structural assessment of product line architecture. Master Thesis on Software Engineering, Thesis no. MSE-2004:24. School of Engineering, Blekinge Institute of Technology, Sweden.Sethi, K., Cai, Y., Wong, S., Garcia, A., & Sant’Anna, C. (2009). From retrospect to prospect: Assessing modularity and stability from software architecture. Joint working IEEE/IFIP conference on software architecture, 2009 & European conference on software architecture. WICSA/ECSA.Shaik, I., Abdelmoez, W,. Gunnalan, R., Shereshevsky, M., Zeid, A., Ammar, H. H., et al. (2005). Change propagation for assessing design quality of software architectures. 5th working IEEE/IFIP conference on software architecture (WICSA’05).Siegmund, N., Rosenmüller, M., Kuhlemann, M., Kästner, C., & Saake, G. (2008). Measuring non-functional properties in software product lines for product derivation. In 15th Asia-Pacific software engineering conference, Beijing, China.Sun Her, J., Hyeok Kim, J., Hun Oh, S., Yul Rhew, S., & Dong Kim, S. (2007). A framework for evaluating reusability of core asset in product line engineering. Information and Software Technology, 49, 740–760.Svahnberg, M., & Bosch, J. (2000). Evolution in software product lines. In 3rd international workshop on software architectures for products families (IWSAPF-3). Las Palmas de Gran Canaria.Van der Hoek, A., Dincel, E., & Medidović, N. (2003). Using services utilization metrics to assess the structure of product line architectures. In 9th international software metrics symposium (METRICS), Sydney, Australia.Van der Linden, F., Schmid, K., & Rommes, E. (2007). Software product lines in action. Springer.Whitmire, S. (1997). Object oriented design measurement. John Wiley & Sons.Wnuk, K., Regnell, B., & Karlsson, L. (2009). What happened to our features? Visualization and understanding of scope change dynamics in a large-scale industrial setting. In 17th IEEE international requirements engineering conference.Yoshimura, K., Ganesan, D., & Muthig, D. (2006). Assessing merge potential of existing engine control systems into a product line. In International workshop on software engineering for automative systems, Shangai, China, pp. 61–67.Zhang, T., Deng, L., Wu, J., Zhou, Q., & Ma, C. (2008). Some metrics for accessing quality of product line architecture. In International conference on computer science and software engineering (CSSE), Wuhan, China, pp. 500–503

Towards guidelines for building a business case and gathering evidence of software reference architectures in industry

Background: Software reference architectures are becoming widely adopted by organizations that need to support the design and maintenance of software applications of a shared domain. For organizations that plan to adopt this architecture-centric approach, it becomes fundamental to know the return on investment and to understand how software reference architectures are designed, maintained, and used. Unfortunately, there is little evidence-based support to help organizations with these challenges. Methods: We have conducted action research in an industry-academia collaboration between the GESSI research group and everis, a multinational IT consulting firm based in Spain. Results: The results from such collaboration are being packaged in order to create guidelines that could be used in similar contexts as the one of everis. The main result of this paper is the construction of empirically-grounded guidelines that support organizations to decide on the adoption of software reference architectures and to gather evidence to improve RA-related practices. Conclusions: The created guidelines could be used by other organizations outside of our industry-academia collaboration. With this goal in mind, we describe the guidelines in detail for their use.Peer ReviewedPostprint (published version

Software Measurement Activities in Small and Medium Enterprises: an Empirical Assessment

An empirical study for evaluating the proper implementation of measurement/metric programs in software companies in one area of Turkey is presented. The research questions are discussed and validated with the help of senior software managers (more than 15 years’ experience) and then used for interviewing a variety of medium and small scale software companies in Ankara. Observations show that there is a common reluctance/lack of interest in utilizing measurements/metrics despite the fact that they are well known in the industry. A side product of this research is that internationally recognized standards such as ISO and CMMI are pursued if they are a part of project/job requirements; without these requirements, introducing those standards to the companies remains as a long-term target to increase quality

SmartUnit: Empirical Evaluations for Automated Unit Testing of Embedded Software in Industry

In this paper, we aim at the automated unit coverage-based testing for embedded software. To achieve the goal, by analyzing the industrial requirements and our previous work on automated unit testing tool CAUT, we rebuild a new tool, SmartUnit, to solve the engineering requirements that take place in our partner companies. SmartUnit is a dynamic symbolic execution implementation, which supports statement, branch, boundary value and MC/DC coverage. SmartUnit has been used to test more than one million lines of code in real projects. For confidentiality motives, we select three in-house real projects for the empirical evaluations. We also carry out our evaluations on two open source database projects, SQLite and PostgreSQL, to test the scalability of our tool since the scale of the embedded software project is mostly not large, 5K-50K lines of code on average. From our experimental results, in general, more than 90% of functions in commercial embedded software achieve 100% statement, branch, MC/DC coverage, more than 80% of functions in SQLite achieve 100% MC/DC coverage, and more than 60% of functions in PostgreSQL achieve 100% MC/DC coverage. Moreover, SmartUnit is able to find the runtime exceptions at the unit testing level. We also have reported exceptions like array index out of bounds and divided-by-zero in SQLite. Furthermore, we analyze the reasons of low coverage in automated unit testing in our setting and give a survey on the situation of manual unit testing with respect to automated unit testing in industry.Comment: In Proceedings of 40th International Conference on Software Engineering: Software Engineering in Practice Track, Gothenburg, Sweden, May 27-June 3, 2018 (ICSE-SEIP '18), 10 page

Influential factors of aligning Spotify squads in mission-critical and offshore projects – a longitudinal embedded case study

Changing the development process of an organization is one of the toughest and riskiest decisions. This is particularly true if the known experiences and practices of the new considered ways of working are relative and subject to contextual assumptions. Spotify engineering culture is deemed as a new agile software development method which increasingly attracts large-scale organizations. The method relies on several small cross-functional self-organized teams (i.e., squads). The squad autonomy is a key driver in Spotify method, where a squad decides what to do and how to do it. To enable effective squad autonomy, each squad shall be aligned with a mission, strategy, short-term goals and other squads. Since a little known about Spotify method, there is a need to answer the question of: How can organizations work out and maintain the alignment to enable loosely coupled and tightly aligned squads? In this paper, we identify factors to support the alignment that is actually performed in practice but have never been discussed before in terms of Spotify method. We also present Spotify Tailoring by highlighting the modified and newly introduced processes to the method. Our work is based on a longitudinal embedded case study which was conducted in a real-world large-scale offshore software intensive organization that maintains mission-critical systems. According to the confidentiality agreement by the organization in question, we are not allowed to reveal a detailed description of the features of the explored project

Relevance, benefits, and problems of software modelling and model driven techniques—A survey in the Italian industry

Context Claimed benefits of software modelling and model driven techniques are improvements in productivity, portability, maintainability and interoperability. However, little effort has been devoted at collecting evidence to evaluate their actual relevance, benefits and usage complications. Goal The main goals of this paper are: (1) assess the diffusion and relevance of software modelling and MD techniques in the Italian industry, (2) understand the expected and achieved benefits, and (3) identify which problems limit/prevent their diffusion. Method We conducted an exploratory personal opinion survey with a sample of 155 Italian software professionals by means of a Web-based questionnaire on-line from February to April 2011. Results Software modelling and MD techniques are very relevant in the Italian industry. The adoption of simple modelling brings common benefits (better design support, documentation improvement, better maintenance, and higher software quality), while MD techniques make it easier to achieve: improved standardization, higher productivity, and platform independence. We identified problems, some hindering adoption (too much effort required and limited usefulness) others preventing it (lack of competencies and supporting tools). Conclusions The relevance represents an important objective motivation for researchers in this area. The relationship between techniques and attainable benefits represents an instrument for practitioners planning the adoption of such techniques. In addition the findings may provide hints for companies and universitie