Feature modelling is a common mechanism for variability management in the context of software product lines. After years of progress, the number of proposals to automatically analyse feature models is still modest and the data about the performance of the different solvers and logic representations used in such area are practically non–existent. Three of the most promising proposals for the automated analysis of feature models are based on the mapping of feature models into CSP, SAT and BDD solvers. in this paper we present a performance test between three off-the-shelf Java CSP, SAT and BDD solvers to analyse feature models which is a novel contribution. in addition, we conclude that the integration of such proposals in a framework will be a key challenge in the future

Benavides Cuevas, David Felipe

Ruiz Cortés, Antonio

Segura Rueda, Sergio

Trinidad Martín Arroyo, Pablo

idUS. Depósito de Investigación Universidad de Sevilla

1A first step towards a framework for the automatedanalysis of feature modelsDavid Benavides, Sergio Segura, Pablo Trinidad, Antonio Ruiz-Corte´sDpto. of Computer Languages and SystemsUniversity of Seville{benavides, sergio, trinidad, aruiz}@tdg.lsi.us.esAbstract— Feature modelling is a common mechanism forvariability management in the context of software product lines.After years of progress, the number of proposals to automaticallyanalyse feature models is still modest and the data about theperformance of the different solvers and logic representationsused in such area are practically non–existent. Three of the mostpromising proposals for the automated analysis of feature modelsare based on the mapping of feature models into CSP, SAT andBDD solvers. In this paper we present a performance test betweenthree off-the-shelf Java CSP, SAT and BDD solvers to analysefeature models which is a novel contribution. In addition, weconclude that the integration of such proposals in a frameworkwill be a key challenge in the future.Index Terms— Software Product Lines, Variability Manage-ment, Feature Models.I. INTRODUCTIONFeature Models (FMs) are one of the most common vari-ability mechanisms. Good tool support is needed to debug,extract information and in summary analyze FMs in order toselect them as a variability mechanism in a Software ProductLine (SPL) approach. A FM represents all possible productsof a SPL in a single model using features. FMs can be used indifferent stages of development such as requirements engineer-ing [10], [11] , architecture definition or code generation [1],[3]. A FM is a tree–like structure and consists of: i)relationsbetween a parent feature and its child features. ii) cross–treeconstraints that are typically inclusion or exclusion statementsof the form “if feature F is included, then feature X must alsobe included (or excluded)”.Automated analysis of FMs is an important challenge inSPL research [1], [2]. It can be performed using off–the–shelf solvers to automatically extract useful information ofthe SPL such as the number of possible combinations offeatures, all the configurations following a criteria, findingthe minimum cost configuration, etc. Although there havebeen some promising proposals based in the representationof FMs as a Constraint Satisfaction Problem (CSP), booleanSATisfiability problem (SAT) and Binary Decision Diagrams(BDD) the performance of the solvers working with suchrepresentations is unknown for the SPL community.In a previous work, we presented a performance comparisonof two CSP java solvers analysing FMs [8]. In this paper we gofurther integrating different solvers and logic representations.First we give a complete mapping for the three solvers (BDD,SAT and CSP) and then we present a performance comparisonof them. To the best of our knowledge, this is the first test thatmeasures the performance of solvers dealing with differentlogic representations of FMs.The remainder of the paper is structured as follows: inSection II the automated analysis of FMs is outlined anddetails on how to translate a FM into a CSP, BDD and SAT arepresented. Section III focuses on the results of the experiment.Finally we summarize our conclusions and describe our futurework in Section IV.II. AUTOMATED ANALYSIS OF FEATURE MODELSOnce a FM is translated into a suitable representation it ispossible to use off–the–shelf solvers to automatically performa great variety of operations such as calculating the numberof possible combinations of features, retrieving configurationsfollowing a criteria, finding the minimum cost configuration,etc [6].There is a great variety of techniques and tools that can beused in the automated analysis of FMs. This paper focus onthree well known problems in the area of automated reasoning:Constraint Satisfaction Problems (CSP), Boolean SatisfiabilityProblems (SAT) and Binary Decision Diagrams (BDD). Allthose representations have not been yet fully adopted in theautomated analysis of FMs. In the next sections we will givea brief overview of each of them and finally we will introducehow translating a FM into a CSP, SAT and BDD.A. Constraint Satisfaction ProblemConstraint Programming can be defined as the set of tech-niques such as algorithms or heuristics that deal with CSPs.A CSP consists on a set of variables, finite domains for thosevariables and a set of constraints restricting the values ofthe variables. A CSP is solved by finding states (values forvariables) in which all constraints are satisfied. CSP solverscan deal with numerical values such as integer domains. Themain ideas concerning the use of constraint programming onFM analysis were stated in [6], [7].B. Boolean Satisfiability Problem (SAT)A propositional formula is an expression consisting on aset of boolean variables (literals) connected by logic opera-tors (¬,∧,∨,→,↔). The propositional satisfiability problem(SAT) consists on deciding whether a given propositional2formula is satisfiable, that is, if logical values can be assignedto its variables in a way that makes the formula true.The problem is restricted by using the propositional formu-las in conjunctive normal form (CNF), that is, propositionalformulas composed by a conjunction of clauses in which eachclause is a disjunction of literals (e.g.((L1∨L2)∧(L3∨L4)∧(L5∨L6))). Every propositional formula can be converted intoan equivalent formula in CNF by using logical equivalences.The basic concepts about the using of SAT in the automatedanalysis of FMs were introduced in [1].C. Binary Decision Diagrams (BDD)A Binary Decision Diagram (BDD) is a data structure usedto represent a boolean function. A BDD is a rooted, directed,acyclic graph composed by a group of decision nodes andtwo terminal nodes called 0-terminal and 1-terminal. Eachnode in the graph represents a variable in a boolean functionand has two child nodes representing an assignment of thevariable to 0 and 1. All paths from the root to the 1-terminalrepresents the variable assignments for which the representedboolean function is true meanwhile all paths to the 0-terminalrepresents the variable assignments for which the representedboolean function is false.Although the size of BDDs can be reduced according tosome established rules, the weakness of this kind of represen-tation is the size of the data structure which may vary betweena linear to an exponential range depending upon the orderingof the variables. Calculating the best variable ordering is anNP-hard problem. In the context of the automated analysis ofFMs there are some tools that claim the internal use of BDDs[9]D. MappingRules for translating FMs to constraints are listed in Figure1. In all cases the notation more common in the bibliographyhas been used. The final representation of the FM is theconjunction of the translated relations following the rules ofFigure 1 plus an additional constraint selecting the root whichis included in all products.III. EXPERIMENTAL RESULTSThe experiments focused on a performance comparison ofthree off–the–shelf Java solvers working with CSP, SAT andBDD in order to test how these representations can influencein the automatic analysis of FMs. The comparison results wereobtained from the execution of a number of FMs mapped asCSP, BDD and SAT in three of the most popular Java solverswithin the research community: JaCoP1 (CSP), JavaBDD2(BDD) and Sat4j3 (SAT).1http://www.cs.lth.se/home/Radoslaw zymanek/2http://javabdd.sourceforge.net3http://www.sat4j.orgN. of Features N. of instances Dependencies[50-100) 50 [0%-25%][100-150) 50 [0%-25%][150-200) 50 [0%-25%][200-300] 50 [0%-25%]TABLE IEXPERIMENTSA. The Experimentwe used four group of 50 randomly generated FMs. Eachgroup included FMs with a number of features in an specificrange ([50-100),[100-150),[150-200) and [200-300)) with adouble aim: test the performance of small, medium and largeinstances and working out averages from the results in orderto avoid as much exogenous interferences as possible. Afterformulating each one as a CSP, BDD and SAT, we proceededwith the execution. Each FM was executed several timesincreasing the number of cross–tree constraints from one untilthe 25% of the number of the features in the FM in orderto find out how dependencies influence in the performance.The dependencies were added randomly as well, but chekingthat the same feature can not appear in more than one cross–tree constraint and that a feature can not have a cross–treeconstraint with any of its ancestors. Averages were obtainedfrom all the FMs in each range with the same percentage ofcross–tree constraints. Table I summarizes the characteristicsof the experiments.As exposed in [6], [7], there are some operations that can beperformed. For our experiments we performed two operations:i) finding out if a model is satisfiable, that is, if it has at lastone solution and ii) finding the total number of configurationsof a given FM. The first one is the simplest operation whilethe second is the hardest one in terms of performance becauseit is necessary to work out the total number of possiblecombinations. The data extracted from the tests were:• Average memory used by the logic representation of theFM (measured in Kilobytes)• Average execution time to find one solution (measured inmilliseconds).• Total number of solutions, that is, the potential numberof products represented in the FM.• Average execution time to obtain the number of solutions(measured in milliseconds).In order to evaluate the implementation, we measured itsperformance and effectiveness. We implemented the solutionusing Java 1.5.0 04. We ran our tests on a WINDOWS XPPROFESSIONAL SP2 machine equipped with a 3Ghz IntelPentium IV microprocessor and 512 MB of DDR RAMmemory.B. The ResultsThe experimental comparison revealed some interestingresults. The first evidence was that JavaBDD is on average96% faster than JaCoP and 75% faster than Sat4j findingone solution. However, JavaBDD revealed a memory usageon average 928% higher than JaCoP and 1672% higher thanSat4j. On the other hand, although JaCoP and Sat4j showed3Fig. 1. Mappinga similar memory usage, SAT representation showed betterresults in both aspects, memory and especially in time. Theperformance of the solvers was similar in the four groups ofexperiments. Figures 2 and 3 presents the results for the groupof FMs with a number of features between 100 and 150.In fact, Figure 2 can be confusing in the sense that in theworst case the memory usage is insignificant (in the order of 2Mb) but this behavior seems to be exponential with the numberof features and dependencies. For instance, in the range of(200-300) features, we found some cases where the memoryused by the solver was around 300 Mb. We think that in biggerFMs (e.g. 1000 features) this can be even a bigger problem.What Figure 2 try to stress is the difference in the use ofmemory of the three solvers.The results obtained from finding the total number ofconfigurations of a given FM showed a great superiorityof JavaBDD. While JaCoP and Sat4j were computationallyincapable of performing that operation in a reasonable time inmost of the cases, JavaBDD lasted 5312 ms to work out the7.77× 1034 solutions of the worst case.Finally, we found some unexpected results or outliers in thedata obtained from the experiments with JaCoP and JavaBDD.On the one hand, JaCoP showed in a few consecutive ex-ecutions a huge number of backtracks and consequently agreat time penalty. On the other hand, JavaBDD revealed ina few experiments a huge memory usage which seemed toincrease exponentially with the number of dependencies. Weare investigating the possible causes of these behaviors [5].C. DiscussionThe great superiority of JavaBDD on finding the totalnumber of solutions is because for calculating the number ofsolutions, in general, CSP and SAT solvers have to retrieve allthe solutions (which is a #P-complete problem [12]) mean-while BDD solvers use efficient graph algorithms to calculatethe total number of solutions without the need of calculatingall the solutions. The huge memory usage of BDD solversdepends on the variable ordering for representing the BDD.40200400600800100012001400160018000 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25Percentage of DependenciesMemory(KB)Memory BDD Memory CSP Memory SATFig. 2. Memory Usage01020304050600 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25Percentage of DependenciesTime(ms)Time BDD Time CSP Time SATFig. 3. Average time to get one solutionAs stated earlier, the size of BDDs can be reduced with agood variable ordering, however, calculating the best variableordering is a NP-hard problem.To the best of our knowledge, there is only a proposal toinclude feature attributes in the automated analysis of featuremodels [6]. This proposal uses CSP solvers for that aim and weare not aware of any result where BDD or SAT solvers couldbe used to deal with feature attributes in order to maximizeor minimize values.As a result of the test, we claim that there is not anoptimum representation for all the possible operations that canbe performed on FMs. Therefore, we think that a frameworkfor the automated analysis of feature models is needed. Theframework will be designed to be open to other solvers whereformal semantics of feature models will play a fundamentalrole [13]. The development of such a framework is the seedof our ongoing research [4].IV. CONCLUSION AND FUTURE WORKIn this paper we integrated the use of different solvers inthe automated analysis of FMs. We presented how to translatea feature model into a CSP, SAT and BDD and we performeda comparative test between three off–the–shelf Java solversmanaging with each representation. The results showed thatusing BDDs for determining satisfiability in a FM is muchfaster than using SAT or CSP. On the other hand, FMs mappedas BDDs required a bigger memory usage in comparison withCSP and SAT. The test also revealed that while FMs mappedas SAT and CSP are computationally incapable of finding thetotal number of configurations in most of medium and largesize FMs, FMs mapped as BDDs can get it in a very low time.We think that there is not an optimum representation forall the possible operations that can be performed on FMs,therefore a framework for the automated analysiss of featuremodels is needed.ACKNOWLEDGEMENTSWe thank Don Batory and Jean-Christophe Trigaux for theirhelpful comments on an earlier draft of this paper.REFERENCES[1] D. Batory. Feature models, grammars, and propositional formulas. InSoftware Product Lines Conference, LNCS 3714, pages 7–20, 2005.[2] D. Batory, D. Benavides, and A. Ruiz-Corte´s. Automated analysisof feature models: Challenges ahead. Communications of the ACM,Conditionally accepted, 2006.[3] D. Batory, J. Sarvela, and A. Rauschmayer. Scaling step-wise refine-ment. IEEE Trans. Software Eng., 30(6):355–371, 2004.[4] D. Benavides, A. Dura´n, A. Ruiz-Corte´s, P. Trinidad, and S. Segura. Aframework for the automated manipulation of software product lines. Inpreparation.5[5] D. Benavides, A. Ruiz-Corte´s, B. Smith, Barry O’Sullivan, andP. Trinidad. Computational issues on the automated analyses of featuremodels using constraint programming. International Journal of SoftwareEngineering and Knowledge Engineering, in preparation, 2006.[6] D. Benavides, A. Ruiz-Corte´s, and P. Trinidad. Automated reasoningon feature models. LNCS, Advanced Information Systems Engineering:17th International Conference, CAiSE 2005, 3520:491–503, 2005.[7] D. Benavides, A. Ruiz-Corte´s, and P. Trinidad. Using constraint pro-gramming to reason on feature models. In The Seventeenth InternationalConference on Software Engineering and Knowledge Engineering, SEKE2005, 2005.[8] D. Benavides, S. Segura, P. Trinidad, and A. Ruiz-Corte´s. Using javacsp solvers in the automated analyses of feature models. LNCS, to beasigned:to be asigned, 2006.[9] K. Czarnecki and P. Kim. Cardinality-based feature modeling andconstraints: A progress report. In Proceedings of the InternationalWorkshop on Software Factories At OOPSLA 2005, 2005.[10] S. Jarzabek, Wai Chun Ong, and Hongyu Zhang. Handling variantrequirements in domain modeling. The Journal of Systems and Software,68(3):171–182, 2003.[11] M. Mannion. Using First-Order Logic for Product Line Model Valida-tion. In Proceedings of the Second Software Product Line Conference(SPLC2), LNCS 2379, pages 176–187, San Diego, CA, 2002. Springer.[12] G. Pesant. Counting solutions of csps: A structural approach. In IJCAI,pages 260–265, 2005.[13] P. Schobbens, P. Heymans, J. Trigaux, and Y. Bontemps. FeatureDiagrams: A Survey and A Formal Semantics. In Proceedings of the14th IEEE International Requirements Engineering Conference (RE’06),Minneapolis, Minnesota, USA, September 2006.

A First Step Towards a Framework for the Automated Analysis of Feature Models

https://idus.us.es/xmlui/bitstream/handle/11441/26361/file_1.pdf?sequence=1&isAllowed=y

A First Step Towards a Framework for the Automated Analysis of Feature Models

Abstract

Similar works

Full text

Available Versions

idUS. Depósito de Investigación Universidad de Sevilla