A systematic review of quality attributes and measures for software product lines

[EN] It is widely accepted that software measures provide an appropriate mechanism for understanding, monitoring, controlling, and predicting the quality of software development projects. In software product lines (SPL), quality is even more important than in a single software product since, owing to systematic reuse, a fault or an inadequate design decision could be propagated to several products in the family. Over the last few years, a great number of quality attributes and measures for assessing the quality of SPL have been reported in literature. However, no studies summarizing the current knowledge about them exist. This paper presents a systematic literature review with the objective of identifying and interpreting all the available studies from 1996 to 2010 that present quality attributes and/or measures for SPL. These attributes and measures have been classified using a set of criteria that includes the life cycle phase in which the measures are applied; the corresponding quality characteristics; their support for specific SPL characteristics (e. g., variability, compositionality); the procedure used to validate the measures, etc. We found 165 measures related to 97 different quality attributes. The results of the review indicated that 92% of the measures evaluate attributes that are related to maintainability. In addition, 67% of the measures are used during the design phase of Domain Engineering, and 56% are applied to evaluate the product line architecture. However, only 25% of them have been empirically validated. In conclusion, the results provide a global vision of the state of the research within this area in order to help researchers in detecting weaknesses, directing research efforts, and identifying new research lines. In particular, there is a need for new measures with which to evaluate both the quality of the artifacts produced during the entire SPL life cycle and other quality characteristics. There is also a need for more validation (both theoretical and empirical) of existing measures. In addition, our results may be useful as a reference guide for practitioners to assist them in the selection or the adaptation of existing measures for evaluating their software product lines. © 2011 Springer Science+Business Media, LLC.This research has been funded by the Spanish Ministry of Science and Innovation under the MULTIPLE (Multimodeling Approach For Quality-Aware Software Product Lines) project with ref. TIN2009-13838.Montagud Gregori, S.; Abrahao Gonzales, SM.; Insfrán Pelozo, CE. (2012). A systematic review of quality attributes and measures for software product lines. Software Quality Journal. 20(3-4):425-486. https://doi.org/10.1007/s11219-011-9146-7S425486203-4Abdelmoez, W., Nassar, D. M., Shereschevsky, M., Gradetsky, N., Gunnalan, R., Ammar, H. H., et al. (2004). Error propagation in software architectures. In 10th international symposium on software metrics (METRICS), Chicago, Illinois, USA.Ajila, S. A., & Dumitrescu, R. T. (2007). Experimental use of code delta, code churn, and rate of change to understand software product line evolution. Journal of Systems and Software, 80, 74–91.Aldekoa, G., Trujillo, S., Sagardui, G., & Díaz, O. (2006). Experience measuring maintainability in software product lines. In XV Jornadas de Ingeniería del Software y Bases de Datos (JISBD). Barcelona.Aldekoa, G., Trujillo, S., Sagardui, G., & Díaz, O. (2008). Quantifying maintanibility in feature oriented product lines, Athens, Greece, pp. 243–247.Alves de Oliveira Junior, E., Gimenes, I. M. S., & Maldonado, J. C. (2008). A metric suite to support software product line architecture evaluation. In XXXIV Conferencia Latinamericana de Informática (CLEI), Santa Fé, Argentina, pp. 489–498.Alves, V., Niu, N., Alves, C., & Valença, G. (2010). Requirements engineering for software product lines: A systematic literature review. Information & Software Technology, 52(8), 806–820.Bosch, J. (2000). Design and use of software architectures: Adopting and evolving a product line approach. USA: ACM Press/Addison-Wesley Publishing Co.Briand, L. C., Differing, C. M., & Rombach, D. (1996a). Practical guidelines for measurement-based process improvement. Software Process-Improvement and Practice, 2, 253–280.Briand, L. C., Morasca, S., & Basili, V. R. (1996b). Property based software engineering measurement. IEEE Transactions on Software Eng., 22(1), 68–86.Calero, C., Ruiz, J., & Piattini, M. (2005). Classifying web metrics using the web quality model. Online Information Review, 29(3): 227–248.Chen, L., Ali Babar, M., & Ali, N. (2009). Variability management in software product lines: A systematic review. In 13th international software product lines conferences (SPLC), San Francisco, USA.Clements, P., & Northrop, L. (2002). Software product lines. 2003. Software product lines practices and patterns. Boston, MA: Addison-Wesley.Crnkovic, I., & Larsson, M. (2004). Classification of quality attributes for predictability in component-based systems. Journal of Econometrics, pp. 231–250.Conference Rankings of Computing Research and Education Association of Australasia (CORE). (2010). Available in http://core.edu.au/index.php/categories/conference%20rankings/1 .Davis, A., Dieste, Ó., Hickey, A., Juristo, N., & Moreno, A. M. (2006). Effectiveness of requirements elicitation techniques: Empirical results derived from a systematic review. In 14th IEEE international conference requirements engineering, pp. 179–188.de Souza Filho, E. D., de Oliveira Cavalcanti, R., Neiva, D. F. S., Oliveira, T. H. B., Barachisio Lisboa, L., de Almeida E. S., & de Lemos Meira, S. R. (2008). Evaluating domain design approaches using systematic review. In 2nd European conference on software architecture, Cyprus, pp. 50–65.Ejiogu, L. (1991). Software engineering with formal metrics. QED Publishing.Engström, E., & Runeson, P. (2011). Software product line testing—A systematic mapping study. Information & Software Technology, 53(1), 2–13.Etxeberria, L., Sagarui, G., & Belategi, L. (2008). Quality aware software product line engineering. Journal of the Brazilian Computer Society, 14(1), Campinas Mar.Ganesan, D., Knodel, J., Kolb, R., Haury, U., & Meier, G. (2007). Comparing costs and benefits of different test strategies for a software product line: A study from Testo AG. In 11th international software product line conference, Kyoto, Japan, pp. 74–83, September 2007.Gómez, O., Oktaba, H., Piattini, M., & García, F. (2006). A systematic review measurement in software engineering: State-of-the-art in measures. In First international conference on software and data technologies (ICSOFT), Setúbal, Portugal, pp. 11–14.IEEE standard for a software quality metrics methodology, IEEE Std 1061-1998, 1998.Inoki, M., & Fukazawa, Y. (2007). Software product line evolution method based on Kaizen approach. In 22nd annual ACM symposium on applied computing, Korea.Insfran, E., & Fernandez, A. (2008). A systematic review of usability evaluation in Web development. 2nd international workshop on web usability and accessibility (IWWUA’08), New Zealand, LNCS 5176, Springer, pp. 81–91.ISO/IEC 25010. (2008). Systems and software engineering. Systems and software Quality Requirements and Evaluation (SQuaRE). System and software quality models.ISO/IEC 9126. (2000). Software engineering. Product Quality.Johansson, E., & Höst, R. (2002). Tracking degradation in software product lines through measurement of design rule violations. In 14th International conference on software engineering and knowledge engineering, Ischia, Italy, pp. 249–254.Journal Citation Reports of Thomson Reuters. (2010). Available in http://thomsonreuters.com/products_services/science/science_products/a-z/journal_citation_reports/ .Khurum, M., & Gorschek, T. (2009). A systematic review of domain analysis solutions for product lines. The Journal of Systems and Software.Kim, T., Ko, I. Y., Kang, S. W., & Lee, D. H. (2008). Extending ATAM to assess product line architecture. In 8th IEEE international conference on computer and information technology, pp. 790–797.Kitchenham, B. (2007). Guidelines for performing systematic literature reviews in software engineering. Version 2.3, EBSE Technical Report, Keele University, UK.Kitchenham, B., Pfleeger, S., & Fenton, N. (1995). Towards a framework for software measurement validation. IEEE Transactions on Software Engineering, 21(12).Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.Mendes, E. (2005). A systematic review of Web engineering research. International symposium on empirical software engineering. Noosa Heads, Australia.Meyer, M. H., & Dalal, D. (2002). Managing platform architectures and manufacturing processes for non assembled products. Journal of Product Innovation Management, 19(4), 277–293.Montagud, S., & Abrahão, S. (2009). Gathering Current knowledge about quality evaluation in software product lines. In 13th international software product lines conferences (SPLC), San Francisco, USA.Montagud, S., & Abrahão, S. (2009). A SQuaRE-bassed quality evaluation method for software product lines. Master’s thesis, December 2009 (in Spanish).Needham, D., & Jones, S. (2006). A software fault tree metric. In 22nd international conference on software maintenance (ICSM), Philadelphia, Pennsylvania, USA.Niemelä, E., & Immonen, A. (2007). Capturing quality requirements of product family architecture. Information and Software Technology, 49(11–12), 1107–1120.Odia, O. E. (2007). Testing in software product lines. Master Thesis Software Engineering of School of Engineering, Bleking Institute of Technology. Thesis no. MSE-2007:16, Sweden.Olumofin, F. G., & Mišić, V. B. (2007). A holistic architecture assessment method for software product lines. Information and Software Technology, 49, 309–323.Pérez Lamancha, B., Polo Usaola, M., & Piattini Velthius, M. (2009). Software product line testing—a systematic review. ICSOFT, (1), 23–30.Poels, G., & Dedene, G. (2000). Distance-based software measurement: necessary and sufficient properties for software measures. Information and Software Technology, 42(I), 35–46.Prehofer, C., van Gurp, J., & Bosch, J. (2008). Compositionality in software platforms. In Emerging methods, technologies and process management in software engineering. Wiley.Rahman, A. (2004). Metrics for the structural assessment of product line architecture. Master Thesis on Software Engineering, Thesis no. MSE-2004:24. School of Engineering, Blekinge Institute of Technology, Sweden.Sethi, K., Cai, Y., Wong, S., Garcia, A., & Sant’Anna, C. (2009). From retrospect to prospect: Assessing modularity and stability from software architecture. Joint working IEEE/IFIP conference on software architecture, 2009 & European conference on software architecture. WICSA/ECSA.Shaik, I., Abdelmoez, W,. Gunnalan, R., Shereshevsky, M., Zeid, A., Ammar, H. H., et al. (2005). Change propagation for assessing design quality of software architectures. 5th working IEEE/IFIP conference on software architecture (WICSA’05).Siegmund, N., Rosenmüller, M., Kuhlemann, M., Kästner, C., & Saake, G. (2008). Measuring non-functional properties in software product lines for product derivation. In 15th Asia-Pacific software engineering conference, Beijing, China.Sun Her, J., Hyeok Kim, J., Hun Oh, S., Yul Rhew, S., & Dong Kim, S. (2007). A framework for evaluating reusability of core asset in product line engineering. Information and Software Technology, 49, 740–760.Svahnberg, M., & Bosch, J. (2000). Evolution in software product lines. In 3rd international workshop on software architectures for products families (IWSAPF-3). Las Palmas de Gran Canaria.Van der Hoek, A., Dincel, E., & Medidović, N. (2003). Using services utilization metrics to assess the structure of product line architectures. In 9th international software metrics symposium (METRICS), Sydney, Australia.Van der Linden, F., Schmid, K., & Rommes, E. (2007). Software product lines in action. Springer.Whitmire, S. (1997). Object oriented design measurement. John Wiley & Sons.Wnuk, K., Regnell, B., & Karlsson, L. (2009). What happened to our features? Visualization and understanding of scope change dynamics in a large-scale industrial setting. In 17th IEEE international requirements engineering conference.Yoshimura, K., Ganesan, D., & Muthig, D. (2006). Assessing merge potential of existing engine control systems into a product line. In International workshop on software engineering for automative systems, Shangai, China, pp. 61–67.Zhang, T., Deng, L., Wu, J., Zhou, Q., & Ma, C. (2008). Some metrics for accessing quality of product line architecture. In International conference on computer science and software engineering (CSSE), Wuhan, China, pp. 500–503

Supporting the automated generation of modular product line safety cases

Abstract The effective reuse of design assets in safety-critical Software Product Lines (SPL) would require the reuse of safety analyses of those assets in the variant contexts of certification of products derived from the SPL. This in turn requires the traceability of SPL variation across design, including variation in safety analysis and safety cases. In this paper, we propose a method and tool to support the automatic generation of modular SPL safety case architectures from the information provided by SPL feature modeling and model-based safety analysis. The Goal Structuring Notation (GSN) safety case modeling notation and its modular extensions supported by the D-Case Editor were used to implement the method in an automated tool support. The tool was used to generate a modular safety case for an automotive Hybrid Braking System SPL

Towards Automated Performance Bug Identification in Python

Context: Software performance is a critical non-functional requirement, appearing in many fields such as mission critical applications, financial, and real time systems. In this work we focused on early detection of performance bugs; our software under study was a real time system used in the advertisement/marketing domain. Goal: Find a simple and easy to implement solution, predicting performance bugs. Method: We built several models using four machine learning methods, commonly used for defect prediction: C4.5 Decision Trees, Na\"{\i}ve Bayes, Bayesian Networks, and Logistic Regression. Results: Our empirical results show that a C4.5 model, using lines of code changed, file's age and size as explanatory variables, can be used to predict performance bugs (recall=0.73, accuracy=0.85, and precision=0.96). We show that reducing the number of changes delivered on a commit, can decrease the chance of performance bug injection. Conclusions: We believe that our approach can help practitioners to eliminate performance bugs early in the development cycle. Our results are also of interest to theoreticians, establishing a link between functional bugs and (non-functional) performance bugs, and explicitly showing that attributes used for prediction of functional bugs can be used for prediction of performance bugs

Automatic allocation of safety requirements to components of a software product line

Safety critical systems developed as part of a product line must still comply with safety standards. Standards use the concept of Safety Integrity Levels (SILs) to drive the assignment of system safety requirements to components of a system under design. However, for a Software Product Line (SPL), the safety requirements that need to be allocated to a component may vary in different products. Variation in design can indeed change the possible hazards incurred in each product, their causes, and can alter the safety requirements placed on individual components in different SPL products. Establishing common SILs for components of a large scale SPL by considering all possible usage scenarios, is desirable for economies of scale, but it also poses challenges to the safety engineering process. In this paper, we propose a method for automatic allocation of SILs to components of a product line. The approach is applied to a Hybrid Braking System SPL design

Towards an aspect weaving BPEL engine

This position paper proposes the use of dynamic aspects and the visitor design pattern to obtain a highly configurable and extensible BPEL engine. Using these two techniques, the core of this infrastructural software can be customised to meet new requirements and add features such as debugging, execution monitoring, or changing to another Web Service selection policy. Additionally, it can easily be extended to cope with customer-specific BPEL extensions. We propose the use of dynamic aspects not only on the engine itself but also on the workflow in order to tackle the problems of Web Service hot deployment and hot fixes to long running processes. In this way, composing aWeb Service "on-the-fly" means weaving its choreography interface into the workflow

Detecting Coordination Problems in Collaborative Software Development Environments

Software development is rarely an individual effort and generally involves teams of developers collaborating to generate good reliable code. Among the software code there exist technical dependencies that arise from software components using services from other components. The different ways of assigning the design, development, and testing of these software modules to people can cause various coordination problems among them. We claim\ud that the collaboration of the developers, designers and testers must be related to and governed by the technical task structure. These collaboration practices are handled in what we call Socio-Technical Patterns.\ud The TESNA project (Technical Social Network Analysis) we report on in this paper addresses this issue. We propose a method and a tool that a project manager can use in order to detect the socio-technical coordination problems. We test the method and tool in a case study of a small and innovative software product company

Enhancing modeling and change support for process families through change patterns

The increasing adoption of process-aware information systems (PAISs), together with the variability of business processes (BPs), has resulted in large collections of related process model variants (i.e., process families). To effectively deal with process families, several proposals (e.g., C-EPC, Provop) exist that extend BP modeling languages with variability-specific constructs. While fostering reuse and reducing modeling efforts, respective constructs imply additional complexity and demand proper support for process designers when creating and modifying process families. Recently, generic and language independent adaptation patterns were successfully introduced for creating and evolving single BP models. However, they are not sufficient to cope with the specific needs for modeling and evolving process families. This paper suggests a complementary set of generic and language-independent change patterns specifically tailored to the needs of process families. When used in combination with existing adaptation patterns, change patterns for process families will enable the modeling and evolution of process families at a high-level of abstraction. Further, they will serve as reference for implementing tools or comparing proposals managing process families. © 2013 Springer-Verlag.This work has been developed with the support of MICINN under the Project EVERYWARE TIN2010-18011.Ayora Esteras, C.; Torres Bosch, MV.; Weber, B.; Reichert, M.; Pelechano Ferragud, V. (2013). Enhancing modeling and change support for process families through change patterns. En Enterprise, Business-Process and Information Systems Modeling, BPMDS 2013. Springer Verlag. 246-260. https://doi.org/10.1007/978-3-642-38484-4_18S246260van der Aalst, W.M.P., ter Hofstede, A.H.M., Barros, B.: Workflow Patterns. Distributed and Parallel Databases 14(1), 5–51 (2003)Aghakasiri, Z., Mirian-Hosseinabadi, S.H.: Workflow change patterns: Opportunities for extension and reuse. In: Proc. SERA 2009, pp. 265–275 (2009)Ayora, C., Torres, V., Reichert, M., Weber, B., Pelechano, V.: Towards run-time flexibility for process families: Open issues and research challenges. In: La Rosa, M., Soffer, P. (eds.) BPM 2012 Workshops. LNBIP, vol. 132, pp. 477–488. Springer, Heidelberg (2013)Ayora, C., Torres, V., Weber, B., Reichert, M., Pelechano, V.: Change patterns for process families. Technical Report, PROS-TR-2012-06, http://www.pros.upv.es/technicalreports/PROS-TR-2012-06.pdfDadam, P., Reichert, M.: The ADEPT project: a decade of research and development for robust and flexible process support. Com. Sci. - R&D 23, 81–97 (2009)Dijkman, R., La Rosa, M., Reijers, H.A.: Managing large collections of business process models - Current techniques and challenges. Comp. in Ind. 63(2), 91–97 (2012)Döhring, M., Zimmermann, B., Karg, L.: Flexible workflows at design- and runtime using BPMN2 adaptation patterns. In: Abramowicz, W. (ed.) BIS 2011. LNBIP, vol. 87, pp. 25–36. Springer, Heidelberg (2011)Gottschalk, F.: Configurable process models. Ph.D. thesis, Eindhoven University of Technology, The Netherlands (2009)Grambow, G., Oberhauser, R., Reichert, M.: Contextual injection of quality measures into software engineering processes. Intl. J. Adv. in Software 4, 76–99 (2011)Gschwind, T., Koehler, J., Wong, J.: Applying patterns during business process modeling. In: Dumas, M., Reichert, M., Shan, M.-C. (eds.) BPM 2008. LNCS, vol. 5240, pp. 4–19. Springer, Heidelberg (2008)Günther, C.W., Rinderle, S., Reichert, M., van der Aalst, W.M.P.: Change mining in adaptive process management systems. In: Meersman, R., Tari, Z. (eds.) OTM 2006. LNCS, vol. 4275, pp. 309–326. Springer, Heidelberg (2006)Hallerbach, A., Bauer, T., Reichert, M.: Context-based configuration of process variants. In: Proc. TCoB 2008, pp. 31–40 (2008)Hallerbach, A., Bauer, T., Reichert, M.: Capturing variability in business process models: the Provop approach. J. of Software Maintenance 22(6-7), 519–546 (2010)Kitchenham, B., Charters, S.: Guidelines for performing Systematic Literature Reviews in Software Engineering, Technical Report EBSE/EPIC–2007–01 (2007)Kulkarni, V., Barat, S., Roychoudhury, S.: Towards business application product lines. In: France, R.B., Kazmeier, J., Breu, R., Atkinson, C. (eds.) MODELS 2012. LNCS, vol. 7590, pp. 285–301. Springer, Heidelberg (2012)Küster, J.M., Gerth, C., Förster, A., Engels, G.: Detecting and resolving process model differences in the absence of a change log. In: Dumas, M., Reichert, M., Shan, M.-C. (eds.) BPM 2008. LNCS, vol. 5240, pp. 244–260. Springer, Heidelberg (2008)Küster, J.M., Gerth, C., Engels, G.: Dynamic computation of change operations in version management of business process models. In: Kühne, T., Selic, B., Gervais, M.-P., Terrier, F. (eds.) ECMFA 2010. LNCS, vol. 6138, pp. 201–216. Springer, Heidelberg (2010)Lanz, A., Weber, B., Reichert, M.: Time patterns for process-aware information systems. Requirements Engineering, 1–29 (2012)La Rosa, M., van der Aalst, W.M.P., Dumas, M., ter Hofstede, A.H.M.: Questionnaire-based variability modeling for system configuration. Software and System Modeling 8(2), 251–274 (2009)Lerner, B.S., Christov, S., Osterweil, L.J., Bendraou, R., Kannengiesser, U., Wise, A.: Exception Handling Patterns for Process Modeling. IEEE Transactions on Software Engineering 36(2), 162–183 (2010)Li, C., Reichert, M., Wombacher, A.: Mining business process variants: Challenges, scenarios, algorithms. Data Knowledge & Engineering 70(5), 409–434 (2011)Marrella, A., Mecella, M., Russo, A.: Featuring automatic adaptivity through workflow enactment and planning. In: Proc. CollaborateCom 2011, pp. 372–381 (2011)Müller, D., Herbst, J., Hammori, M., Reichert, M.: IT support for release management processes in the automotive industry. In: Dustdar, S., Fiadeiro, J.L., Sheth, A.P. (eds.) BPM 2006. LNCS, vol. 4102, pp. 368–377. Springer, Heidelberg (2006)Reichert, M., Weber, B.: Enabling flexibility in process-aware information systems: challenges, methods, technologies. Springer (2012)Reinhartz-Berger, I., Soffer, P., Sturm, A.: Organizational reference models: supporting an adequate design of local business processes. IBPIM 4(2), 134–149 (2009)Rosemann, M., van der Aalst, W.M.P.: A configurable reference modeling language. Information Systems 32(1), 1–23 (2007)Russell, N., ter Hofstede, A.H.M., Edmond, D., van der Aalst, W.M.P.: Workflow data patterns. Technical Report FIT-TR-2004-01, Queensland Univ. of Technology (2004)Russell, N., ter Hofstede, A.H.M., Edmond, D., van der Aalst, W.M.P.: Workflow resource patterns. Technical Report WP 127, Eindhoven Univ. of Technology (2004)Russell, N., van der Aalst, W.M.P., ter Hofstede, A.H.M.: Workflow Exception Patterns. In: Martinez, F.H., Pohl, K. (eds.) CAiSE 2006. LNCS, vol. 4001, pp. 288–302. Springer, Heidelberg (2006)Smirnov, S., Weidlich, M., Mendling, J., Weske, M.: Object-sensitive action patterns in process model repositories. In: Muehlen, M.z., Su, J. (eds.) BPM 2010 Workshops. LNBIP, vol. 66, pp. 251–263. Springer, Heidelberg (2011)Weber, B., Reichert, M., Rinderle-Ma, S.: Change patterns and change support features - Enhancing flexibility in process-aware information systems. Data Knowledge & Engineering 66, 438–466 (2008)Weber, B., Sadiq, S., Reichert, M.: Beyond rigidity - dynamic process lifecycle support. Computer Science 23, 47–65 (2009)Weber, B., Reichert, M., Reijers, H.A., Mendling, J.: Refactoring large process model repositories. Computers in Industry 62(5), 467–486 (2011