A systematic review of quality attributes and measures for software product lines

[EN] It is widely accepted that software measures provide an appropriate mechanism for understanding, monitoring, controlling, and predicting the quality of software development projects. In software product lines (SPL), quality is even more important than in a single software product since, owing to systematic reuse, a fault or an inadequate design decision could be propagated to several products in the family. Over the last few years, a great number of quality attributes and measures for assessing the quality of SPL have been reported in literature. However, no studies summarizing the current knowledge about them exist. This paper presents a systematic literature review with the objective of identifying and interpreting all the available studies from 1996 to 2010 that present quality attributes and/or measures for SPL. These attributes and measures have been classified using a set of criteria that includes the life cycle phase in which the measures are applied; the corresponding quality characteristics; their support for specific SPL characteristics (e. g., variability, compositionality); the procedure used to validate the measures, etc. We found 165 measures related to 97 different quality attributes. The results of the review indicated that 92% of the measures evaluate attributes that are related to maintainability. In addition, 67% of the measures are used during the design phase of Domain Engineering, and 56% are applied to evaluate the product line architecture. However, only 25% of them have been empirically validated. In conclusion, the results provide a global vision of the state of the research within this area in order to help researchers in detecting weaknesses, directing research efforts, and identifying new research lines. In particular, there is a need for new measures with which to evaluate both the quality of the artifacts produced during the entire SPL life cycle and other quality characteristics. There is also a need for more validation (both theoretical and empirical) of existing measures. In addition, our results may be useful as a reference guide for practitioners to assist them in the selection or the adaptation of existing measures for evaluating their software product lines. © 2011 Springer Science+Business Media, LLC.This research has been funded by the Spanish Ministry of Science and Innovation under the MULTIPLE (Multimodeling Approach For Quality-Aware Software Product Lines) project with ref. TIN2009-13838.Montagud Gregori, S.; Abrahao Gonzales, SM.; Insfrán Pelozo, CE. (2012). A systematic review of quality attributes and measures for software product lines. Software Quality Journal. 20(3-4):425-486. https://doi.org/10.1007/s11219-011-9146-7S425486203-4Abdelmoez, W., Nassar, D. M., Shereschevsky, M., Gradetsky, N., Gunnalan, R., Ammar, H. H., et al. (2004). Error propagation in software architectures. In 10th international symposium on software metrics (METRICS), Chicago, Illinois, USA.Ajila, S. A., & Dumitrescu, R. T. (2007). Experimental use of code delta, code churn, and rate of change to understand software product line evolution. Journal of Systems and Software, 80, 74–91.Aldekoa, G., Trujillo, S., Sagardui, G., & Díaz, O. (2006). Experience measuring maintainability in software product lines. In XV Jornadas de Ingeniería del Software y Bases de Datos (JISBD). Barcelona.Aldekoa, G., Trujillo, S., Sagardui, G., & Díaz, O. (2008). Quantifying maintanibility in feature oriented product lines, Athens, Greece, pp. 243–247.Alves de Oliveira Junior, E., Gimenes, I. M. S., & Maldonado, J. C. (2008). A metric suite to support software product line architecture evaluation. In XXXIV Conferencia Latinamericana de Informática (CLEI), Santa Fé, Argentina, pp. 489–498.Alves, V., Niu, N., Alves, C., & Valença, G. (2010). Requirements engineering for software product lines: A systematic literature review. Information & Software Technology, 52(8), 806–820.Bosch, J. (2000). Design and use of software architectures: Adopting and evolving a product line approach. USA: ACM Press/Addison-Wesley Publishing Co.Briand, L. C., Differing, C. M., & Rombach, D. (1996a). Practical guidelines for measurement-based process improvement. Software Process-Improvement and Practice, 2, 253–280.Briand, L. C., Morasca, S., & Basili, V. R. (1996b). Property based software engineering measurement. IEEE Transactions on Software Eng., 22(1), 68–86.Calero, C., Ruiz, J., & Piattini, M. (2005). Classifying web metrics using the web quality model. Online Information Review, 29(3): 227–248.Chen, L., Ali Babar, M., & Ali, N. (2009). Variability management in software product lines: A systematic review. In 13th international software product lines conferences (SPLC), San Francisco, USA.Clements, P., & Northrop, L. (2002). Software product lines. 2003. Software product lines practices and patterns. Boston, MA: Addison-Wesley.Crnkovic, I., & Larsson, M. (2004). Classification of quality attributes for predictability in component-based systems. Journal of Econometrics, pp. 231–250.Conference Rankings of Computing Research and Education Association of Australasia (CORE). (2010). Available in http://core.edu.au/index.php/categories/conference%20rankings/1 .Davis, A., Dieste, Ó., Hickey, A., Juristo, N., & Moreno, A. M. (2006). Effectiveness of requirements elicitation techniques: Empirical results derived from a systematic review. In 14th IEEE international conference requirements engineering, pp. 179–188.de Souza Filho, E. D., de Oliveira Cavalcanti, R., Neiva, D. F. S., Oliveira, T. H. B., Barachisio Lisboa, L., de Almeida E. S., & de Lemos Meira, S. R. (2008). Evaluating domain design approaches using systematic review. In 2nd European conference on software architecture, Cyprus, pp. 50–65.Ejiogu, L. (1991). Software engineering with formal metrics. QED Publishing.Engström, E., & Runeson, P. (2011). Software product line testing—A systematic mapping study. Information & Software Technology, 53(1), 2–13.Etxeberria, L., Sagarui, G., & Belategi, L. (2008). Quality aware software product line engineering. Journal of the Brazilian Computer Society, 14(1), Campinas Mar.Ganesan, D., Knodel, J., Kolb, R., Haury, U., & Meier, G. (2007). Comparing costs and benefits of different test strategies for a software product line: A study from Testo AG. In 11th international software product line conference, Kyoto, Japan, pp. 74–83, September 2007.Gómez, O., Oktaba, H., Piattini, M., & García, F. (2006). A systematic review measurement in software engineering: State-of-the-art in measures. In First international conference on software and data technologies (ICSOFT), Setúbal, Portugal, pp. 11–14.IEEE standard for a software quality metrics methodology, IEEE Std 1061-1998, 1998.Inoki, M., & Fukazawa, Y. (2007). Software product line evolution method based on Kaizen approach. In 22nd annual ACM symposium on applied computing, Korea.Insfran, E., & Fernandez, A. (2008). A systematic review of usability evaluation in Web development. 2nd international workshop on web usability and accessibility (IWWUA’08), New Zealand, LNCS 5176, Springer, pp. 81–91.ISO/IEC 25010. (2008). Systems and software engineering. Systems and software Quality Requirements and Evaluation (SQuaRE). System and software quality models.ISO/IEC 9126. (2000). Software engineering. Product Quality.Johansson, E., & Höst, R. (2002). Tracking degradation in software product lines through measurement of design rule violations. In 14th International conference on software engineering and knowledge engineering, Ischia, Italy, pp. 249–254.Journal Citation Reports of Thomson Reuters. (2010). Available in http://thomsonreuters.com/products_services/science/science_products/a-z/journal_citation_reports/ .Khurum, M., & Gorschek, T. (2009). A systematic review of domain analysis solutions for product lines. The Journal of Systems and Software.Kim, T., Ko, I. Y., Kang, S. W., & Lee, D. H. (2008). Extending ATAM to assess product line architecture. In 8th IEEE international conference on computer and information technology, pp. 790–797.Kitchenham, B. (2007). Guidelines for performing systematic literature reviews in software engineering. Version 2.3, EBSE Technical Report, Keele University, UK.Kitchenham, B., Pfleeger, S., & Fenton, N. (1995). Towards a framework for software measurement validation. IEEE Transactions on Software Engineering, 21(12).Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.Mendes, E. (2005). A systematic review of Web engineering research. International symposium on empirical software engineering. Noosa Heads, Australia.Meyer, M. H., & Dalal, D. (2002). Managing platform architectures and manufacturing processes for non assembled products. Journal of Product Innovation Management, 19(4), 277–293.Montagud, S., & Abrahão, S. (2009). Gathering Current knowledge about quality evaluation in software product lines. In 13th international software product lines conferences (SPLC), San Francisco, USA.Montagud, S., & Abrahão, S. (2009). A SQuaRE-bassed quality evaluation method for software product lines. Master’s thesis, December 2009 (in Spanish).Needham, D., & Jones, S. (2006). A software fault tree metric. In 22nd international conference on software maintenance (ICSM), Philadelphia, Pennsylvania, USA.Niemelä, E., & Immonen, A. (2007). Capturing quality requirements of product family architecture. Information and Software Technology, 49(11–12), 1107–1120.Odia, O. E. (2007). Testing in software product lines. Master Thesis Software Engineering of School of Engineering, Bleking Institute of Technology. Thesis no. MSE-2007:16, Sweden.Olumofin, F. G., & Mišić, V. B. (2007). A holistic architecture assessment method for software product lines. Information and Software Technology, 49, 309–323.Pérez Lamancha, B., Polo Usaola, M., & Piattini Velthius, M. (2009). Software product line testing—a systematic review. ICSOFT, (1), 23–30.Poels, G., & Dedene, G. (2000). Distance-based software measurement: necessary and sufficient properties for software measures. Information and Software Technology, 42(I), 35–46.Prehofer, C., van Gurp, J., & Bosch, J. (2008). Compositionality in software platforms. In Emerging methods, technologies and process management in software engineering. Wiley.Rahman, A. (2004). Metrics for the structural assessment of product line architecture. Master Thesis on Software Engineering, Thesis no. MSE-2004:24. School of Engineering, Blekinge Institute of Technology, Sweden.Sethi, K., Cai, Y., Wong, S., Garcia, A., & Sant’Anna, C. (2009). From retrospect to prospect: Assessing modularity and stability from software architecture. Joint working IEEE/IFIP conference on software architecture, 2009 & European conference on software architecture. WICSA/ECSA.Shaik, I., Abdelmoez, W,. Gunnalan, R., Shereshevsky, M., Zeid, A., Ammar, H. H., et al. (2005). Change propagation for assessing design quality of software architectures. 5th working IEEE/IFIP conference on software architecture (WICSA’05).Siegmund, N., Rosenmüller, M., Kuhlemann, M., Kästner, C., & Saake, G. (2008). Measuring non-functional properties in software product lines for product derivation. In 15th Asia-Pacific software engineering conference, Beijing, China.Sun Her, J., Hyeok Kim, J., Hun Oh, S., Yul Rhew, S., & Dong Kim, S. (2007). A framework for evaluating reusability of core asset in product line engineering. Information and Software Technology, 49, 740–760.Svahnberg, M., & Bosch, J. (2000). Evolution in software product lines. In 3rd international workshop on software architectures for products families (IWSAPF-3). Las Palmas de Gran Canaria.Van der Hoek, A., Dincel, E., & Medidović, N. (2003). Using services utilization metrics to assess the structure of product line architectures. In 9th international software metrics symposium (METRICS), Sydney, Australia.Van der Linden, F., Schmid, K., & Rommes, E. (2007). Software product lines in action. Springer.Whitmire, S. (1997). Object oriented design measurement. John Wiley & Sons.Wnuk, K., Regnell, B., & Karlsson, L. (2009). What happened to our features? Visualization and understanding of scope change dynamics in a large-scale industrial setting. In 17th IEEE international requirements engineering conference.Yoshimura, K., Ganesan, D., & Muthig, D. (2006). Assessing merge potential of existing engine control systems into a product line. In International workshop on software engineering for automative systems, Shangai, China, pp. 61–67.Zhang, T., Deng, L., Wu, J., Zhou, Q., & Ma, C. (2008). Some metrics for accessing quality of product line architecture. In International conference on computer science and software engineering (CSSE), Wuhan, China, pp. 500–503

A Quality Model for Actionable Analytics in Rapid Software Development

Background: Accessing relevant data on the product, process, and usage perspectives of software as well as integrating and analyzing such data is crucial for getting reliable and timely actionable insights aimed at continuously managing software quality in Rapid Software Development (RSD). In this context, several software analytics tools have been developed in recent years. However, there is a lack of explainable software analytics that software practitioners trust. Aims: We aimed at creating a quality model (called Q-Rapids quality model) for actionable analytics in RSD, implementing it, and evaluating its understandability and relevance. Method: We performed workshops at four companies in order to determine relevant metrics as well as product and process factors. We also elicited how these metrics and factors are used and interpreted by practitioners when making decisions in RSD. We specified the Q-Rapids quality model by comparing and integrating the results of the four workshops. Then we implemented the Q-Rapids tool to support the usage of the Q-Rapids quality model as well as the gathering, integration, and analysis of the required data. Afterwards we installed the Q-Rapids tool in the four companies and performed semi-structured interviews with eight product owners to evaluate the understandability and relevance of the Q-Rapids quality model. Results: The participants of the evaluation perceived the metrics as well as the product and process factors of the Q-Rapids quality model as understandable. Also, they considered the Q-Rapids quality model relevant for identifying product and process deficiencies (e.g., blocking code situations). Conclusions: By means of heterogeneous data sources, the Q-Rapids quality model enables detecting problems that take more time to find manually and adds transparency among the perspectives of system, process, and usage.Comment: This is an Author's Accepted Manuscript of a paper to be published by IEEE in the 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) 2018. The final authenticated version will be available onlin

Hybrid Algorithms Based on Integer Programming for the Search of Prioritized Test Data in Software Product Lines

In Software Product Lines (SPLs) it is not possible, in general, to test all products of the family. The number of products denoted by a SPL is very high due to the combinatorial explosion of features. For this reason, some coverage criteria have been proposed which try to test at least all feature interactions without the necessity to test all products, e.g., all pairs of features (pairwise coverage). In addition, it is desirable to first test products composed by a set of priority features. This problem is known as the Prioritized Pairwise Test Data Generation Problem. In this work we propose two hybrid algorithms using Integer Programming (IP) to generate a prioritized test suite. The first one is based on an integer linear formulation and the second one is based on a integer quadratic (nonlinear) formulation. We compare these techniques with two state-of-the-art algorithms, the Parallel Prioritized Genetic Solver (PPGS) and a greedy algorithm called prioritized-ICPL. Our study reveals that our hybrid nonlinear approach is clearly the best in both, solution quality and computation time. Moreover, the nonlinear variant (the fastest one) is 27 and 42 times faster than PPGS in the two groups of instances analyzed in this work.Universidad de Málaga. Campus de Excelencia Internacional Andalucía Tech. Partially funded by the Spanish Ministry of Economy and Competitiveness and FEDER under contract TIN2014-57341-R, the University of Málaga, Andalucía Tech and the Spanish Network TIN2015-71841-REDT (SEBASENet)

Automated generation of computationally hard feature models using evolutionary algorithms

This is the post-print version of the final paper published in Expert Systems with Applications. The published article is available from the link below. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. Copyright @ 2014 Elsevier B.V.A feature model is a compact representation of the products of a software product line. The automated extraction of information from feature models is a thriving topic involving numerous analysis operations, techniques and tools. Performance evaluations in this domain mainly rely on the use of random feature models. However, these only provide a rough idea of the behaviour of the tools with average problems and are not sufficient to reveal their real strengths and weaknesses. In this article, we propose to model the problem of finding computationally hard feature models as an optimization problem and we solve it using a novel evolutionary algorithm for optimized feature models (ETHOM). Given a tool and an analysis operation, ETHOM generates input models of a predefined size maximizing aspects such as the execution time or the memory consumption of the tool when performing the operation over the model. This allows users and developers to know the performance of tools in pessimistic cases providing a better idea of their real power and revealing performance bugs. Experiments using ETHOM on a number of analyses and tools have successfully identified models producing much longer executions times and higher memory consumption than those obtained with random models of identical or even larger size.European Commission (FEDER), the Spanish Government and the Andalusian Government

Proceedings of the Eighth Annual Software Engineering Workshop

The four major topics of discussion included: the NASA Software Engineering Laboratory, software testing, human factors in software engineering and software quality assessment. As in the past years, there were 12 position papers presented (3 for each topic) followed by questions and very heavy participation by the general audience

Annotated bibliography of software engineering laboratory literature

An annotated bibliography of technical papers, documents, and memorandums produced by or related to the Software Engineering Laboratory is given. More than 100 publications are summarized. These publications cover many areas of software engineering and range from research reports to software documentation. This document has been updated and reorganized substantially since the original version (SEL-82-006, November 1982). All materials have been grouped into eight general subject areas for easy reference: the Software Engineering Laboratory; the Software Engineering Laboratory-software development documents; software tools; software models; software measurement; technology evaluations; Ada technology; and data collection. Subject and author indexes further classify these documents by specific topic and individual author