Using genetic programming to evolve action selection rules in traversal-based automated software testing: results obtained with the TESTAR tool

[EN] Traversal-based automated software testing involves testing an application via its graphical user interface (GUI) and thereby taking the user's point of view and executing actions in a human-like manner. These actions are decided on the fly, as the software under test (SUT) is being run, as opposed to being set up in the form of a sequence prior to the testing, a sequence that is then used to exercise the SUT. In practice, random choice is commonly used to decide which action to execute at each state (a procedure commonly referred to as monkey testing), but a number of alternative mechanisms have also been proposed in the literature. Here we propose using genetic programming (GP) to evolve such an action selection strategy, defined as a list of IF-THEN rules. Genetic programming has proved to be suited for evolving all sorts of programs, and rules in particular, provided adequate primitives (functions and terminals) are defined. These primitives must aim to extract the most relevant information from the SUT and the dynamics of the testing process. We introduce a number of such primitives suited to the problem at hand and evaluate their usefulness based on various metrics. We carry out experiments and compare the results with those obtained by random selection and also by Q-learning, a reinforcement learning technique. Three applications are used as Software Under Test (SUT) in the experiments. The analysis shows the potential of GP to evolve action selection strategies.Esparcia Alcázar, AI.; Almenar-Pedrós, F.; Vos, TE.; Rueda Molina, U. (2018). Using genetic programming to evolve action selection rules in traversal-based automated software testing: results obtained with the TESTAR tool. Memetic Computing. 10(3):257-265. https://doi.org/10.1007/s12293-018-0263-8S257265103Aho P, Menz N, Rty T (2013) Dynamic reverse engineering of GUI models for testing. In: Proceedings of 2013 international conference on control, decision and information technologies (CoDIT’13)Aho P, Oliveira R, Algroth E, Vos T (2016) Evolution of automated testing of software systems through graphical user interface. In: Procs. of the 1st international conference on advances in computation, communications and services (ACCSE 2016), Valencia, pp 16–21Alegroth E, Feldt R, Ryrholm L (2014) Visual GUI testing in practice: challenges, problems and limitations. Empir Softw Eng 20:694–744. https://doi.org/10.1007/s10664-013-9293-5Barr ET, Harman M, McMinn P, Shahbaz M, Yoo S (2015) The oracle problem in software testing: a survey. IEEE Trans Softw Eng 41(5):507–525Bauersfeld S, Vos TEJ (2012) A reinforcement learning approach to automated GUI robustness testing. In: Fast abstracts of the 4th symposium on search-based software engineering (SSBSE 2012), pp 7–12Bauersfeld S, de Rojas A, Vos T (2014) Evaluating rogue user testing in industry: an experience report. In: 2014 IEEE eighth international conference on research challenges in information science (RCIS), pp 1–10. https://doi.org/10.1109/RCIS.2014.6861051Bauersfeld S, Vos TEJ, Condori-Fernández N, Bagnato A, Brosse E (2014) Evaluating the TESTAR tool in an industrial case study. In: 2014 ACM-IEEE international symposium on empirical software engineering and measurement, ESEM 2014, Torino, Italy, September 18–19, 2014, p 4Bauersfeld S, Wappler S, Wegener J (2011) A metaheuristic approach to test sequence generation for applications with a GUI. In: Cohen MB, Ó Cinnéide M (eds) Search based software engineering: third international symposium, SSBSE 2011, Szeged, Hungary, September 10-12, 2011. Proceedings. Springer Berlin Heidelberg, Berlin, Heidelberg, pp 173–187Brameier MF, Banzhaf W (2010) Linear genetic programming, 1st edn. Springer, New YorkChaudhary N, Sangwan O (2016) Metrics for event driven software. Int J Adv Comput Sci Appl 7(1):85–89Esparcia-Alcázar AI, Almenar F, Martínez M, Rueda U, Vos TE (2016) Q-learning strategies for action selection in the TESTAR automated testing tool. In: Proceedings of META 2016 6th international conference on metaheuristics and nature inspired computing, pp 174–180Esparcia-Alcázar AI, Almenar F, Rueda U, Vos TEJ (2017) Evolving rules for action selection in automated testing via genetic programming–a first approach. In: Squillero G, Sim K (eds) Applications of evolutionary computation: 20th European conference, evoapplications 2017, Amsterdam, The Netherlands, April 19–21, 2017, Proceedings, part II. Springer, pp 82–95. https://doi.org/10.1007/978-3-319-55792-2_6Esparcia-Alcázar AI, Moravec J (2013) Fitness approximation for bot evolution in genetic programming. Soft Comput 17(8):1479–1487. https://doi.org/10.1007/s00500-012-0965-7He W, Zhao R, Zhu Q (2015) Integrating evolutionary testing with reinforcement learning for automated test generation of object-oriented software. Chin J Electron 24(1):38–45Koza JR (1992) Genetic programming: on the programming of computers by means of natural selection. MIT Press, CambridgeLehman J, Stanley KO (2011) Novelty search and the problem with objectives. In: Riolo R, Vladislavleva E, Moore JH (eds) Genetic programming theory and practice IX, genetic and evolutionary computation. Springer, New York, pp 37–56Memon AM, Soffa ML, Pollack ME (2001) Coverage criteria for GUI testing. In: Proceedings of ESEC/FSE 2001, pp 256–267Rueda U, Vos TEJ, Almenar F, Martínez MO, Esparcia-Alcázar AI (2015) TESTAR: from academic prototype towards an industry-ready tool for automated testing at the user interface level. In: Canos JH, Gonzalez Harbour M (eds) Actas de las XX Jornadas de Ingeniería del Software y Bases de Datos (JISBD 2015), pp 236–245Seesing A, Gross HG (2006) A genetic programming approach to automated test generation for object-oriented software. Int Trans Syst Sci Appl 1(2):127–134Vos TE, Kruse PM, Condori-Fernández N, Bauersfeld S, Wegener J (2015) TESTAR: tool support for test automation at the user interface level. Int J Inf Syst Model Des 6(3):46–83. https://doi.org/10.4018/IJISMD.2015070103Wappler S, Wegener J (2006) Evolutionary unit testing of object-oriented software using strongly-typed genetic programming. In: Proceedings of the 8th annual conference on genetic and evolutionary computation, GECCO’06. ACM, New York, NY, USA, pp 1925–1932. URL https://doi.org/10.1145/1143997.1144317Watkins C (1989) Learning from delayed rewards. Ph.D. Thesis. Cambridge Universit

Algoritmos baseados em estimadores de distribuição e técnicas da otimização com muitos objetivos aplicados na refatoração automática de software

Search Based Software Engineering (SBSE) is a research area in which optimization algorithms are applied to solve Software Engineering problems. SBSE area comprises several sub-areas, among them, Search Based Software Refactoring (SBSR), which deals with automatic software refactoring processes. One of the problems investigated in SBSR is the determination of a sequence of refactorings that provides code improvement, according to predetermined criteria. To model this problem as an optimization problem, it is necessary to define objective functions. In this case, the objective functions are quality metrics used in Software Engineering. Since there are many metrics to consider, the problem of determining sequences of refactorings is a Many-Objective Combinatorial Optimization Problem. Many-Objective optimization comprises a set of algorithms and techniques used to solve problems that take into account more than three objectives. In order to solve problems of this nature, new methods are proposed to reduce the deterioration of the search as the number of objectives increases, among them, the multi-objective evolutionary algorithms, which include the Estimation of Distribution Algorithms (EDA). However, despite having good results in combinatorial optimization problems the investigation of EDA in the context of optimization with many objectives are still incipient. The goal of this work is to investigate the use of EDA in the automatic selection of a sequence of refactorings. In order to achieve this goal, a new EDA is proposed, which incorporates many-objective optimization techniques existing in literature. The proposed algorithm was validated and added to a previously implemented automatic software refactoring framework. Furthermore, a set of experiments was conducted to evaluate the proposed algorithm. The results obtained by the proposed algorithm were compared to the results obtained by algorithms existing in the literature.A Engenharia de Software Baseada em Busca, conhecida como SBSE (do inglês, Search Based Software Engineering), é uma área que usa algoritmos de otimização para solucionar problemas da Engenharia de Software. A área de SBSE compreende diversas subáreas, dentre estas, encontra-se a Refatoração de Software Baseada em Busca (SBSR, do inglês Search Based Software Refactoring), que trata de processos de refatoração automática de software utilizando algoritmos de otimização. Um dos problemas encontrados na SBSR é a determinação de sequências de refatorações com o objetivo da melhoria do código, de acordo com critérios previamente determinados. Para modelar este problema como um problema de otimização é necessário que sejam definidas funções objetivo. Nesse caso, as funções objetivo são métricas de qualidade utilizadas na Engenharia de Software. Como são muitas as métricas a se considerar, o problema de determinação de sequências de refatorações é naturalmente um Problema de Otimização combinatório com Muitos Objetivos. A otimização com muitos objetivos compreende um conjunto de algoritmos e técnicas que buscam resolver problemas de otimização com mais de três funções objetivo. Para resolver problemas dessa natureza, são propostos novos métodos que visam reduzir a deterioração da busca à medida que o número de objetivos aumenta. Dentre eles, destacam-se os algoritmos evolucionários multiobjetivo, que incluem os Estimation of Distribution Algorithms (EDA). No entanto, apesar de ter bons resultados em problemas de otimização combinatória, os EDA ainda são pouco explorados no contexto da otimização com muitos objetivos. O objetivo deste trabalho é investigar o uso dos EDA na seleção automática de uma sequência de refatorações. Para que esse objetivo seja atingido, propõe-se um novo EDA, no qual são incorporadas técnicas de otimização de muitos objetivos existentes na literatura. O algoritmo proposto foi validado e adicionado a um framework de refatoração automática de software previamente implementado. Após a condução de um conjunto de experimentos, os resultados obtidos pelo algoritmo proposto foram comparados aos resultados obtidos pelos algoritmos encontrados na literatura aplicados no contexto de SBSR.São Cristóvão, S

A User-aware Intelligent Refactoring for Discrete and Continuous Software Integration

Successful software products evolve through a process of continual change. However, this process may weaken the design of the software and make it unnecessarily complex, leading to significantly reduced productivity and increased fault-proneness. Refactoring improves the software design while preserving overall functionality and behavior, and is an important technique in managing the growing complexity of software systems. Most of the existing work on software refactoring uses either an entirely manual or a fully automated approach. Manual refactoring is time-consuming, error-prone and unsuitable for large-scale, radical refactoring. Furthermore, fully automated refactoring yields a static list of refactorings which, when applied, leads to a new and often hard to comprehend design. In addition, it is challenging to merge these refactorings with other changes performed in parallel by developers. In this thesis, we propose a refactoring recommendation approach that dynamically adapts and interactively suggests refactorings to developers and takes their feedback into consideration. Our approach uses Non-dominated Sorting Genetic Algorithm (NSGAII) to find a set of good refactoring solutions that improve software quality while minimizing the deviation from the initial design. These refactoring solutions are then analyzed to extract interesting common features between them such as the frequently occurring refactorings in the best non-dominated solutions. We combined our interactive approach and unsupervised learning to reduce the developer’s interaction effort when refactoring a system. The unsupervised learning algorithm clusters the different trade-off solutions, called the Pareto front, to guide the developers in selecting their region of interests and reduce the number of refactoring options to explore. To reduce the interaction effort, we propose an approach to convert multi-objective search into a mono-objective one after interacting with the developer to identify a good refactoring solution based on their preferences. Since developers may want to focus on specific code locations, the ”Decision Space” is also important. Therefore, our interactive approach enables developers to pinpoint their preference simultaneously in the objective (quality metrics) and decision (code location) spaces. Due to an urgent need for refactoring tools that can support continuous integration and some recent development processes such as DevOps that are based on rapid releases, we propose, for the first time, an intelligent software refactoring bot, called RefBot. Our bot continuously monitors the software repository and find the best sequence of refactorings to fix the quality issues in Continous Integration/Continous Development (CI/CD) environments as a set of pull-requests generated after mining previous code changes to understand the profile of developers. We quantitatively and qualitatively evaluated the performance and effectiveness of our proposed approaches via a set of studies conducted with experienced developers who used our tools on both open source and industry projects.Ph.D.College of Engineering & Computer ScienceUniversity of Michigan-Dearbornhttps://deepblue.lib.umich.edu/bitstream/2027.42/154775/1/Vahid Alizadeh Final Dissertation.pdfDescription of Vahid Alizadeh Final Dissertation.pdf : Dissertatio