A systematic review of quality attributes and measures for software product lines

[EN] It is widely accepted that software measures provide an appropriate mechanism for understanding, monitoring, controlling, and predicting the quality of software development projects. In software product lines (SPL), quality is even more important than in a single software product since, owing to systematic reuse, a fault or an inadequate design decision could be propagated to several products in the family. Over the last few years, a great number of quality attributes and measures for assessing the quality of SPL have been reported in literature. However, no studies summarizing the current knowledge about them exist. This paper presents a systematic literature review with the objective of identifying and interpreting all the available studies from 1996 to 2010 that present quality attributes and/or measures for SPL. These attributes and measures have been classified using a set of criteria that includes the life cycle phase in which the measures are applied; the corresponding quality characteristics; their support for specific SPL characteristics (e. g., variability, compositionality); the procedure used to validate the measures, etc. We found 165 measures related to 97 different quality attributes. The results of the review indicated that 92% of the measures evaluate attributes that are related to maintainability. In addition, 67% of the measures are used during the design phase of Domain Engineering, and 56% are applied to evaluate the product line architecture. However, only 25% of them have been empirically validated. In conclusion, the results provide a global vision of the state of the research within this area in order to help researchers in detecting weaknesses, directing research efforts, and identifying new research lines. In particular, there is a need for new measures with which to evaluate both the quality of the artifacts produced during the entire SPL life cycle and other quality characteristics. There is also a need for more validation (both theoretical and empirical) of existing measures. In addition, our results may be useful as a reference guide for practitioners to assist them in the selection or the adaptation of existing measures for evaluating their software product lines. © 2011 Springer Science+Business Media, LLC.This research has been funded by the Spanish Ministry of Science and Innovation under the MULTIPLE (Multimodeling Approach For Quality-Aware Software Product Lines) project with ref. TIN2009-13838.Montagud Gregori, S.; Abrahao Gonzales, SM.; Insfrán Pelozo, CE. (2012). A systematic review of quality attributes and measures for software product lines. Software Quality Journal. 20(3-4):425-486. https://doi.org/10.1007/s11219-011-9146-7S425486203-4Abdelmoez, W., Nassar, D. M., Shereschevsky, M., Gradetsky, N., Gunnalan, R., Ammar, H. H., et al. (2004). Error propagation in software architectures. In 10th international symposium on software metrics (METRICS), Chicago, Illinois, USA.Ajila, S. A., & Dumitrescu, R. T. (2007). Experimental use of code delta, code churn, and rate of change to understand software product line evolution. Journal of Systems and Software, 80, 74–91.Aldekoa, G., Trujillo, S., Sagardui, G., & Díaz, O. (2006). Experience measuring maintainability in software product lines. In XV Jornadas de Ingeniería del Software y Bases de Datos (JISBD). Barcelona.Aldekoa, G., Trujillo, S., Sagardui, G., & Díaz, O. (2008). Quantifying maintanibility in feature oriented product lines, Athens, Greece, pp. 243–247.Alves de Oliveira Junior, E., Gimenes, I. M. S., & Maldonado, J. C. (2008). A metric suite to support software product line architecture evaluation. In XXXIV Conferencia Latinamericana de Informática (CLEI), Santa Fé, Argentina, pp. 489–498.Alves, V., Niu, N., Alves, C., & Valença, G. (2010). Requirements engineering for software product lines: A systematic literature review. Information & Software Technology, 52(8), 806–820.Bosch, J. (2000). Design and use of software architectures: Adopting and evolving a product line approach. USA: ACM Press/Addison-Wesley Publishing Co.Briand, L. C., Differing, C. M., & Rombach, D. (1996a). Practical guidelines for measurement-based process improvement. Software Process-Improvement and Practice, 2, 253–280.Briand, L. C., Morasca, S., & Basili, V. R. (1996b). Property based software engineering measurement. IEEE Transactions on Software Eng., 22(1), 68–86.Calero, C., Ruiz, J., & Piattini, M. (2005). Classifying web metrics using the web quality model. Online Information Review, 29(3): 227–248.Chen, L., Ali Babar, M., & Ali, N. (2009). Variability management in software product lines: A systematic review. In 13th international software product lines conferences (SPLC), San Francisco, USA.Clements, P., & Northrop, L. (2002). Software product lines. 2003. Software product lines practices and patterns. Boston, MA: Addison-Wesley.Crnkovic, I., & Larsson, M. (2004). Classification of quality attributes for predictability in component-based systems. Journal of Econometrics, pp. 231–250.Conference Rankings of Computing Research and Education Association of Australasia (CORE). (2010). Available in http://core.edu.au/index.php/categories/conference%20rankings/1 .Davis, A., Dieste, Ó., Hickey, A., Juristo, N., & Moreno, A. M. (2006). Effectiveness of requirements elicitation techniques: Empirical results derived from a systematic review. In 14th IEEE international conference requirements engineering, pp. 179–188.de Souza Filho, E. D., de Oliveira Cavalcanti, R., Neiva, D. F. S., Oliveira, T. H. B., Barachisio Lisboa, L., de Almeida E. S., & de Lemos Meira, S. R. (2008). Evaluating domain design approaches using systematic review. In 2nd European conference on software architecture, Cyprus, pp. 50–65.Ejiogu, L. (1991). Software engineering with formal metrics. QED Publishing.Engström, E., & Runeson, P. (2011). Software product line testing—A systematic mapping study. Information & Software Technology, 53(1), 2–13.Etxeberria, L., Sagarui, G., & Belategi, L. (2008). Quality aware software product line engineering. Journal of the Brazilian Computer Society, 14(1), Campinas Mar.Ganesan, D., Knodel, J., Kolb, R., Haury, U., & Meier, G. (2007). Comparing costs and benefits of different test strategies for a software product line: A study from Testo AG. In 11th international software product line conference, Kyoto, Japan, pp. 74–83, September 2007.Gómez, O., Oktaba, H., Piattini, M., & García, F. (2006). A systematic review measurement in software engineering: State-of-the-art in measures. In First international conference on software and data technologies (ICSOFT), Setúbal, Portugal, pp. 11–14.IEEE standard for a software quality metrics methodology, IEEE Std 1061-1998, 1998.Inoki, M., & Fukazawa, Y. (2007). Software product line evolution method based on Kaizen approach. In 22nd annual ACM symposium on applied computing, Korea.Insfran, E., & Fernandez, A. (2008). A systematic review of usability evaluation in Web development. 2nd international workshop on web usability and accessibility (IWWUA’08), New Zealand, LNCS 5176, Springer, pp. 81–91.ISO/IEC 25010. (2008). Systems and software engineering. Systems and software Quality Requirements and Evaluation (SQuaRE). System and software quality models.ISO/IEC 9126. (2000). Software engineering. Product Quality.Johansson, E., & Höst, R. (2002). Tracking degradation in software product lines through measurement of design rule violations. In 14th International conference on software engineering and knowledge engineering, Ischia, Italy, pp. 249–254.Journal Citation Reports of Thomson Reuters. (2010). Available in http://thomsonreuters.com/products_services/science/science_products/a-z/journal_citation_reports/ .Khurum, M., & Gorschek, T. (2009). A systematic review of domain analysis solutions for product lines. The Journal of Systems and Software.Kim, T., Ko, I. Y., Kang, S. W., & Lee, D. H. (2008). Extending ATAM to assess product line architecture. In 8th IEEE international conference on computer and information technology, pp. 790–797.Kitchenham, B. (2007). Guidelines for performing systematic literature reviews in software engineering. Version 2.3, EBSE Technical Report, Keele University, UK.Kitchenham, B., Pfleeger, S., & Fenton, N. (1995). Towards a framework for software measurement validation. IEEE Transactions on Software Engineering, 21(12).Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.Mendes, E. (2005). A systematic review of Web engineering research. International symposium on empirical software engineering. Noosa Heads, Australia.Meyer, M. H., & Dalal, D. (2002). Managing platform architectures and manufacturing processes for non assembled products. Journal of Product Innovation Management, 19(4), 277–293.Montagud, S., & Abrahão, S. (2009). Gathering Current knowledge about quality evaluation in software product lines. In 13th international software product lines conferences (SPLC), San Francisco, USA.Montagud, S., & Abrahão, S. (2009). A SQuaRE-bassed quality evaluation method for software product lines. Master’s thesis, December 2009 (in Spanish).Needham, D., & Jones, S. (2006). A software fault tree metric. In 22nd international conference on software maintenance (ICSM), Philadelphia, Pennsylvania, USA.Niemelä, E., & Immonen, A. (2007). Capturing quality requirements of product family architecture. Information and Software Technology, 49(11–12), 1107–1120.Odia, O. E. (2007). Testing in software product lines. Master Thesis Software Engineering of School of Engineering, Bleking Institute of Technology. Thesis no. MSE-2007:16, Sweden.Olumofin, F. G., & Mišić, V. B. (2007). A holistic architecture assessment method for software product lines. Information and Software Technology, 49, 309–323.Pérez Lamancha, B., Polo Usaola, M., & Piattini Velthius, M. (2009). Software product line testing—a systematic review. ICSOFT, (1), 23–30.Poels, G., & Dedene, G. (2000). Distance-based software measurement: necessary and sufficient properties for software measures. Information and Software Technology, 42(I), 35–46.Prehofer, C., van Gurp, J., & Bosch, J. (2008). Compositionality in software platforms. In Emerging methods, technologies and process management in software engineering. Wiley.Rahman, A. (2004). Metrics for the structural assessment of product line architecture. Master Thesis on Software Engineering, Thesis no. MSE-2004:24. School of Engineering, Blekinge Institute of Technology, Sweden.Sethi, K., Cai, Y., Wong, S., Garcia, A., & Sant’Anna, C. (2009). From retrospect to prospect: Assessing modularity and stability from software architecture. Joint working IEEE/IFIP conference on software architecture, 2009 & European conference on software architecture. WICSA/ECSA.Shaik, I., Abdelmoez, W,. Gunnalan, R., Shereshevsky, M., Zeid, A., Ammar, H. H., et al. (2005). Change propagation for assessing design quality of software architectures. 5th working IEEE/IFIP conference on software architecture (WICSA’05).Siegmund, N., Rosenmüller, M., Kuhlemann, M., Kästner, C., & Saake, G. (2008). Measuring non-functional properties in software product lines for product derivation. In 15th Asia-Pacific software engineering conference, Beijing, China.Sun Her, J., Hyeok Kim, J., Hun Oh, S., Yul Rhew, S., & Dong Kim, S. (2007). A framework for evaluating reusability of core asset in product line engineering. Information and Software Technology, 49, 740–760.Svahnberg, M., & Bosch, J. (2000). Evolution in software product lines. In 3rd international workshop on software architectures for products families (IWSAPF-3). Las Palmas de Gran Canaria.Van der Hoek, A., Dincel, E., & Medidović, N. (2003). Using services utilization metrics to assess the structure of product line architectures. In 9th international software metrics symposium (METRICS), Sydney, Australia.Van der Linden, F., Schmid, K., & Rommes, E. (2007). Software product lines in action. Springer.Whitmire, S. (1997). Object oriented design measurement. John Wiley & Sons.Wnuk, K., Regnell, B., & Karlsson, L. (2009). What happened to our features? Visualization and understanding of scope change dynamics in a large-scale industrial setting. In 17th IEEE international requirements engineering conference.Yoshimura, K., Ganesan, D., & Muthig, D. (2006). Assessing merge potential of existing engine control systems into a product line. In International workshop on software engineering for automative systems, Shangai, China, pp. 61–67.Zhang, T., Deng, L., Wu, J., Zhou, Q., & Ma, C. (2008). Some metrics for accessing quality of product line architecture. In International conference on computer science and software engineering (CSSE), Wuhan, China, pp. 500–503

Feature-based generation of pervasive systems architectures utilizing software product line concepts

As the need for pervasive systems tends to increase and to dominate the computing discipline, software engineering approaches must evolve at a similar pace to facilitate the construction of such systems in an efficient manner. In this thesis, we provide a vision of a framework that will help in the construction of software product lines for pervasive systems by devising an approach to automatically generate architectures for this domain. Using this framework, designers of pervasive systems will be able to select a set of desired system features, and the framework will automatically generate architectures that support the presence of these features. Our approach will not compromise the quality of the architecture especially as we have verified that by comparing the generated architectures to those manually designed by human architects. As an initial step, and in order to determine the most commonly required features that comprise the widely most known pervasive systems, we surveyed more than fifty existing architectures for pervasive systems in various domains. We captured the most essential features along with the commonalities and variabilities between them. The features were categorized according to the domain and the environment that they target. Those categories are: General pervasive systems, domain-specific, privacy, bridging, fault-tolerance and context-awareness. We coupled the identified features with well-designed components, and connected the components based on the initial features selected by a system designer to generate an architecture. We evaluated our generated architectures against architectures designed by human architects. When metrics such as coupling, cohesion, complexity, reusability, adaptability, modularity, modifiability, packing density, and average interaction density were used to test our framework, our generated architectures were found comparable, if not better than the human generated architectures

Variability-Modelling Practices in Industrial Software Product Lines: A Qualitative Study

Many organizations have transitioned from single-systems development to product-line development with the goal of increasing productivity and facilitating mass customization. Variability modelling is a key activity in software product-line development that deals with the explicit representation of variability using dedicated models. Variability models specify points of variability and their variants in a product line. Although many variability-modelling notations and tools have been designed by researchers and practitioners, very little is known about their usage, actual benefits or challenges. Existing studies mostly describe product-line practices in general, with little focus on variability modelling. We address this gap through a qualitative study on variability-modelling practices in medium- and large-scale companies using two empirical methods: surveys and interviews. We investigated companies' variability-modelling practices and experiences with the aim to gather information on 1) the methods and strategies used to create and manage variability models, 2) the tools and notations used for variability modelling, 3) the perceived values and challenges of variability modelling, and 4) the core characteristics of their variability models. Our results show that variability models are often created by re-engineering existing products into a product line. All of the interviewees and the majority of survey participants indicated that they represent variability using separate variability models rather than annotative approaches. We found that developers use variability models for many purposes, such as the visualization of variabilities, configuration of products, and scoping of products. Although we observed that high degree of heterogeneity exists in the variability-modelling notations and tools used by organizations, feature-based notations and tools are the most common. We saw huge differences in the sizes of variability models and their contents, which indicate that variability models can have different use cases depending on the organization. Most of our study participants reported complexity challenges that were related mainly to the visualization and evolution of variability models, and dependency management. In addition, reports from interviews suggest that product-line adoption and variability modelling have forced developers to think in terms of a product-line scenario rather than a product-based scenario

Haasteet ja muunneltavuus ohjelmistotuoteperheiden rakentamisessa: rautateiden sähköenergian selvitysjärjestelmä

According to European Union, all member countries shall have a settlement system in use by 2020. The settlement system shall receive energy consumption data from meters installed in trains, validate the data and allocate it for the right user. In this way, the energy consumption can be invoiced from the right user precisely. Erex is a such energy settlement system. The system needs to be adopted for each country to meet their different needs with regards to laws, systems and practices. This means that the system shall allow at least some flexibility. When new partners have entered the partnership and new instances have been created and modified for them with ad-hoc methods, the manageability of the systems has decreased. For this reason, a need to improve the management of the systems as whole has been raised. It would be easier, if the systems would have a shared core and systematically managed variability. This would mean creating a product family with systematically managed commonality and variability. The objective of this thesis was to study, what are the challenges of creating such product family, where all systems share the same principles but some degree of flexibility is allowed. To achieve these objectives, experts from partner countries and the administration and developers of the systems were interviewed. Thereafter, challenges related to product families and their variability were studied from the literature. Then, the challenges found in empirical and theoretical parts were compared. The objective was to see if the results of empirical study support the current literature. The comparison had three key results. Firstly, many of the current challenges are rather typical for software that is derived with ad-hoc methods. These challenges were found both in empirical and theoretical parts. Secondly, there were a group of challenges that were found only in the theoretical part and did not appear in the interviews but were considered as potential for this case. Thus, these challenges can be of great worth when the product family is developed. Lastly, there were challenges discovered only in the empirical part. These challenges are highly case and domain specific and were not investigated in the theoretical part due to their subjects. Experience from domain should be used to address these case specific challenges as they may not be found from any literature. There were only three challenges that could have been addressed in theoretical part by their subject. Compared to the whole amount of challenges found, these three challenges had only little role. Overall, this means that challenges found in the case are rather typical for product families. Thus, experience from the literature and industry can be used to solve these challenges

The wrong Wright stuff : mapping human error in aviation

The Aviation Safety Reporting System (ASRS) was instituted to aid the Federal Aviation Administration in tracking trends in aviation incidents so that, ultimately, safety measures and training could be implemented to decrease the occurrence of accidents and incidents within the industry. The current system relies on hand coding of reports to recognize current trends and alert the proper parties. Although the filing party may enter some codified data describing the surrounding scenario (e.g., time of day, weather), there is no opportunity to specify a category if the problem is human error. Considering the prevalence of human error within these incidents (around 55% based on a report by Boeing, 2006), a greater understanding of the driving factors is needed. The current study was an investigation of the human error components of airline incident reports. Text analysis tools were applied to ASRS incident narrative reports to determine a classification based on human performance for commercial and general aviation. The results from the current study demonstrate that an empirically based approach can be used to uncover latent categories within the Flight Crew Human Performance\u27 classified reports. The combined approach of latent semantic analysis, k-means clustering, and keyword analysis were used successfully in developing a nine element classification of commercial aviation reports and twelve element classification of general aviation reports. The taxonomies suggested by the current study for both commercial and general aviation reveal categories beyond just human error elements. The classification scheme suggested for the commercial aviation reports most closely resembled the ACCERS taxonomy developed by Krokos and Baker (2005; see also Baker & Krokos, 2007), which was constructed to help in categorizing all incident reports. The classification suggested for general aviation reports did not closely resemble any existing classification scheme. Although the suggested taxonomy shared categories such as situational awareness and communication with classifications such as crew resource management (CRM) or single pilot resource management (SRM), the current classification also holds non-human elements such as weather and context. The taxonomies for both commercial and general aviation revealed a category for context, and the difficulty of flying into certain airports was apparent. These findings can be implemented to improve training programs by assisting in the creation of contextually based training scenarios. Furthermore, based on findings for general aviation in particular, pilots could benefit from increased training in situational awareness and monitoring of notices and airspace

ModelVars2SPL : an automated approach to reengineer model variants into software product lines

Orientadora : Profª. Drª. Silvia R. VergilioCoorientador : Prof Dr. Roberto E. Lopez-HerrejonTese (doutorado) - Universidade Federal do Paraná, Setor de Ciências Exatas, Programa de Pós-Graduação em Informática. Defesa: Curitiba, 11/04/2017Inclui referências : f. 74-82Área de concentração : Ciência da computaçãoResumo: Linhas de Produto de Software (LPSs) são famílias de sistemas de software relacionados que são desenvolvidos para um segmento de mercado ou domínio. Comumente, LPSs surgem de um conjunto de variantes existentes, quando a manutenção e/ou evolução individuais tornam-se complexas. Contudo, as abordagens encontradas na literatura para extração de LPS a partir de variantes existentes não dão suporte a modelos de projeto, são parcialmente automatizadas, ou não refletem restrições de domínio em termos de combinação de características. Para lidar com estas limitações, o objetivo deste trabalho é apresentar uma abordagem automatizada para fazer a reengenharia de variantes de modelos em uma LPS, chamada ModelVars2SPL (Variantes de Modelos para Linha de Produto de Software, do Inglês Model Variants to Software Product Line). A entrada para a abordagem é um conjunto de diagramas de classe Linguagem de Modelagem Unificada (UML) e uma lista de características que estes implementam. Todo o processo de reengenharia é coberto, e a saída inclui (i) um Modelo de Características, que representa a combinação de características das variantes de entrada, e (ii) uma Arquitetura de Linha de Produto, que representa uma arquitetura global com características anotadas. O processo de reengenharia da ModelVars2SPL é composto por quatro passos, sendo dois deles apoiados em técnicas baseadas em busca, e os dois outros baseados em algoritmos determinísticos. Não existe a necessidade de especialistas humanos para obter soluções. Para avaliar a abordagem proposta, foi conduzido um experimento para aferir a qualidade das soluções obtidas. A qualidade dos Modelos de Características e das Arquiteturas de Linha de Produto foi medida considerando-se o quão bem as variantes de entrada foram representadas. Além disso, a qualidade das saídas em cada passo da abordagem foi avaliada levando-se em consideração os objetivos do processo de reengenharia. Para a experimentação utilizaram-se dez estudos de caso representando dois cenários diferentes. Os resultados da avaliação mostram que a abordagem consegue obter soluções com alto grau de corretude em termos de representação das variantes de entrada, e que as saídas dos passos estão de acordo com as fases do processo de reengenharia. Com base em um exemplo de uso de uma solução mostra-se como os artefatos de LPS obtidos facilitam a atividade de manutenção. Palavras-chave: Reúso, Reengenharia, Linha de Produto de Software, Extração de LPS, Engenharia de Software Baseada em Busca.Abstract: Software Product Lines (SPLs) are families of related software systems developed for specific market segments or domains. SPLs commonly emerge from sets of existing variants when their individual maintenance and/or evolution become complex. However, current approaches for SPL extraction from existing variants do not support design models, are partially automated, or do not reflect domain constraints in terms of feature combinations. To tackle these limitations, the goal of this work is to present an automated approach to reengineer model variants into an SPL, called ModelVars2SPL (Model Variants to Software Product Line). The input of the approach is a set of Unified Modeling Language (UML) class diagrams and the list of features they implement. All the reengineering process is covered, and the output includes (i) a Feature Model, which represents the combinations of features of the input variants, and (ii) a Product Line Architecture, which represents a global architecture with feature-related annotations. The reengineering process of ModelVars2SPL is composed of four steps, two of them rely on searchbased techniques and the others are based on deterministic algorithms. There is no need for human experts for obtaining solutions. We conducted an experiment to evaluate the quality of the solutions obtained with the proposed approach. The quality of the FMs and PLAs was measured by considering how well these artifacts represent the input variants. Furthermore, we evaluate the quality of the outputs in each step of the approach taking into account the goals of the reengineering process. For the experimentation we used ten case studies representing two di_erent scenarios. The results of the evaluation show that the approach can obtain solutions with high degree of correctness in terms of representing the input variants, and that the outputs of the steps are in accordance to the phases of the reengineering process. Based on an example of use we show how the obtained FM and PLA make easier the maintenance activity. Keywords: Reuse, Reengineering, Software Product Line, SPL extraction, Search-Based Software Engineering