An Analysis on the Applicability of Meta-Heuristic Searching Techniques for Automated Test Data Generation in Automatic Programming Assessment

حظي تقييم البرمجة التلقائي (APA) بالكثير من الاهتمام بين الباحثين بشكل أساسي لدعم الدرجات الآلية ووضع علامات على المهامالمكلف بادائها الطلاب أو التدريبات بشكل منهجي. يتم تعريف APA بشكل شائع كطريقة يمكن أن تعزز الدقة والكفاءة والاتساق وكذلك تقديمملاحظات فورية لحلول للطلاب. في تحقيق APA ، تعد عملية إنشاء بيانات الاختبار مهمة للغاية وذلك لإجراء اختبار ديناميكي لمهمةالطلاب. في مجال اختبار البرمجيات ، أوضحت العديد من الأبحاث التي تركز على توليد بيانات الاختبار نجاح اعتماد تقنيات البحث الفوقية(MHST) من أجل تعزيز إجراءات استنباط بيانات الاختبار المناسبة للاختبار الفعال. ومع ذلك، فإن الأبحاث التي أجريت على APA حتىالآن لم تستغل بعد التقنيات المفيدة لتشمل تغطية اختبار جودة برنامج أفضل. لذلك ، أجرت هذه الدراسة تقييماً مقارنا لتحديد أي تقنية بحثفوقي قابلة للتطبيق لدعم كفاءة توليد بيانات الاختبار الآلي (ATDG) في تنفيذ اختبار وظيفي ديناميكي. في تقييم البرمجة التلقائي يتم تضمينالعديد من تقنيات البحث الفوقية الحديثة في التقييم المقارن الذي يجمع بين كل من خوارزميات البحث المحلية والعالمية من عام 2000 حتىعام 2018 .تشير نتيجة هذه الدراسة إلى أن تهجين Cuckoo Search مع Tabu Search و lévy flight كواحدة من طرق البحث الفوقية الواعدةليتم تطبيقها ، حيث أنه يتفوق على الطرق الفوقية الأخرى فيما يتعلق بعدد التكرارات ونطاق المدخلات.Automatic Programming Assessment (APA) has been gaining lots of attention among researchers mainly to support automated grading and marking of students’ programming assignments or exercises systematically. APA is commonly identified as a method that can enhance accuracy, efficiency and consistency as well as providing instant feedback on students’ programming solutions. In achieving APA, test data generation process is very important so as to perform a dynamic testing on students’ assignment. In software testing field, many researches that focus on test data generation have demonstrated the successful of adoption of Meta-Heuristic Search Techniques (MHST) so as to enhance the procedure of deriving adequate test data for efficient testing. Nonetheless, thus far the researches on APA have not yet usefully exploited the techniques accordingly to include a better quality program testing coverage. Therefore, this study has conducted a comparative evaluation to identify any applicable MHST to support efficient Automated Test Data Generation (ATDG) in executing a dynamic-functional testing in APA. Several recent MHST are included in the comparative evaluation combining both the local and global search algorithms ranging from the year of 2000 until 2018. Result of this study suggests that the hybridization of Cuckoo Search with Tabu Search and lévy flight as one of promising MHST to be applied, as it’s outperforms other MHST with regards to number of iterations and range of inputs

Comparative evaluation of genetic algorithm-based test case optimization

Software testing is a crucial phase in software development process although it consumes more time and cost of software development. Researchers have proposed several approaches focusing on helping software testers to reduce the execution time and cost of the testing process. Test case optimization is a multi-objective approach that has become one of the best solutions to overcome these problems. Test case optimization focusing on reducing the number of test cases in the test suite that may reduce the overall testing time, cost and effort of software testers especially in regression testing. This paper presents the comparative evaluation between test case optimization techniques that are based on Genetic Algorithm (GA). The evaluation is based on five criteria i.e. technique objectives, applied fitness function, contributions, the percentage of the reduced test cases, fault detection capability, and technique limitations. The evaluation results able identify the gaps in the existing GAbased test case optimization approaches and provide insight in determining the potential research directions in this area.Keywords: Test case optimization, regression testing, multi-objectives, genetic algorithm, software testin

Construction of Prioritized T-Way Test Suite Using Bi-Objective Dragonfly Algorithm

Software testing is important for ensuring the reliability of software systems. In software testing, effective test case generation is essential as an alternative to exhaustive testing. For improving the software testing technology, the t-way testing technique combined with metaheuristic algorithm has been great to analyze a large number of combinations for getting optimal solutions. However, most of the existing t-way strategies consider test case weights while generating test suites. Priority of test cases hasn’t been fully considered in previous works, but in practice, it’s frequently necessary to distinguish between high-priority and low-priority test cases. Therefore, the significance of test case prioritization is quite high. For this reason, this paper has proposed a t-way strategy that implements an adaptive Dragonfly Algorithm (DA) to construct prioritized t-way test suites. Both test case weight and test case priority have equal significance during test suite generation in this strategy. We have designed and implemented a Bi-objective Dragonfly Algorithm (BDA) for prioritized t-way test suite generation, and the two objectives are test case weight and test case priority. The test results demonstrate that BDA performs competitively against existing t-way strategies in terms of test suite size, and in addition, BDA generates prioritized test suites.©2022 Authors. Published by IEEE. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. For more information, see https://creativecommons.org/licenses/by-nc-nd/4.0/fi=vertaisarvioitu|en=peerReviewed

An analysis on the applicability of meta-heuristic searching techniques for automated test data generation in automatic programming assessment

Automatic Programming Assessment (APA) has been gaining lots of attention among researchers mainly to support automated grading and marking of students’ programming assignments or exercises systematically. APA is commonly identified as a method that can enhance accuracy, efficiency and consistency as well as providing instant feedback on students’ programming solutions. In achieving APA, test data generation process is very important so as to perform a dynamic testing on students’ assignment. In software testing field, many researches that focus on test data generation have demonstrated the successful of adoption of Meta-Heuristic Search Techniques (MHST) so as to enhance the procedure of deriving adequate test data for efficient testing. Nonetheless, thus far the researches on APA have not yet usefully exploited the techniques accordingly to include a better quality program testing coverage. Therefore, this study has conducted a comparative evaluation to identify any applicable MHST to support efficient Automated Test Data Generation (ATDG) in executing a dynamic-functional testing in APA. Several recent MHST are included in the comparative evaluation combining both the local and global search algorithms ranging from the year of 2000 until 2018. Result of this study suggests that the hybridization of Cuckoo Search with Tabu Search and lévy flight as one of promising MHST to be applied, as it’s outperforms other MHST with regards to number of iterations and range of inputs

The design and applications of the african buffalo algorithm for general optimization problems

Optimization, basically, is the economics of science. It is concerned with the need to maximize profit and minimize cost in terms of time and resources needed to execute a given project in any field of human endeavor. There have been several scientific investigations in the past several decades on discovering effective and efficient algorithms to providing solutions to the optimization needs of mankind leading to the development of deterministic algorithms that provide exact solutions to optimization problems. In the past five decades, however, the attention of scientists has shifted from the deterministic algorithms to the stochastic ones since the latter have proven to be more robust and efficient, even though they do not guarantee exact solutions. Some of the successfully designed stochastic algorithms include Simulated Annealing, Genetic Algorithm, Ant Colony Optimization, Particle Swarm Optimization, Bee Colony Optimization, Artificial Bee Colony Optimization, Firefly Optimization etc. A critical look at these ‘efficient’ stochastic algorithms reveals the need for improvements in the areas of effectiveness, the number of several parameters used, premature convergence, ability to search diverse landscapes and complex implementation strategies. The African Buffalo Optimization (ABO), which is inspired by the herd management, communication and successful grazing cultures of the African buffalos, is designed to attempt solutions to the observed shortcomings of the existing stochastic optimization algorithms. Through several experimental procedures, the ABO was used to successfully solve benchmark optimization problems in mono-modal and multimodal, constrained and unconstrained, separable and non-separable search landscapes with competitive outcomes. Moreover, the ABO algorithm was applied to solve over 100 out of the 118 benchmark symmetric and all the asymmetric travelling salesman’s problems available in TSPLIB95. Based on the successful experimentation with the novel algorithm, it is safe to conclude that the ABO is a worthy contribution to the scientific literature

Optimization-Based Evolutionary Data Mining Techniques for Structural Health Monitoring

In recent years, data mining technology has been employed to solve various Structural Health Monitoring (SHM) problems as a comprehensive strategy because of its computational capability. Optimization is one the most important functions in Data mining. In an engineering optimization problem, it is not easy to find an exact solution. In this regard, evolutionary techniques have been applied as a part of procedure of achieving the exact solution. Therefore, various metaheuristic algorithms have been developed to solve a variety of engineering optimization problems in SHM. This study presents the most applicable as well as effective evolutionary techniques used in structural damage identification. To this end, a brief overview of metaheuristic techniques is discussed in this paper. Then the most applicable optimization-based algorithms in structural damage identification are presented, i.e. Particle Swarm Optimization (PSO), Genetic Algorithm (GA), Imperialist Competitive Algorithm (ICA) and Ant Colony Optimization (ACO). Some related examples are also detailed in order to indicate the efficiency of these algorithms

Bio-inspired optimization algorithms for unit test generation

Tese de Mestrado, Engenharia Informática (Engenharia de Software), 2021, Universidade de Lisboa, Faculdade de CiênciasNa sociedade atual nós estamos rodeados e usamos todo o tipo de aplicações de software. Problemas no software pode causar todo o tipo de consequências, desde pessoas não conseguirem jogar um jogo como era suposto a uma aeronave despenhar-se matando toda as pessoas a bordo. De modo a que se evite certas consequências, convém que esse software não tenha problemas e funcione como é suposto. Porém, o software é escrito por humanos pelo que está sujeito a ter erros. Para lidar com esta situação, testes de software são feitos, de modo a que se descubra e resolva os problemas no software. Testar software baseado em pesquisa é uma área de teste de software que se tem mostrado bastante bemsucedida na geração de conjuntos de teste unitários otimizados para cobertura de código. Esta abordagem usa algoritmos meta-heurísticos guiados por critérios de cobertura de código para gerar os testes. Neste estudo, foi utilizado um critério de cobertura múltiplo que é composto por oito critérios diferentes: a cobertura de linhas, cobertura de ramos, cobertura de métodos, cobertura de métodos de nível de topo sem exceção, cobertura de ramos direto, cobertura de output, mutação fraca e cobertura de exceções. No que diz respeito aos algoritmos meta-heurísticos, os algoritmos evolucionários são o estado da arte atual, tendo apresentado os melhores resultados em estudos anteriores, superando os algoritmos aleatórios. No entanto, serão os algoritmos evolucionários realmente os melhores algoritmos neste contexto? E quanto aos algoritmos de inteligência de grupo, poderão eles também apresentar bons resultados? Poderá o atual estado da arte ser substituído por um algoritmo de inteligência de grupo? Deste modo, para responder a estas e outras questões, decidimos explorar os algoritmos bio-inspirados, também conhecidos por algoritmos de inteligência de grupo. Estes algoritmos baseiam-se no comportamento de indivíduos que pertencem a grupos na natureza, tais como os enxames de abelhas. Os algoritmos bio-inspirados não são completamente novos na área de testar software. Estudos anteriores mostram que os algoritmos de inteligência de grupo são geralmente melhores que os algoritmos genéticos para testes de estrutura, que na geração de dados para testes o desempenho dos algoritmos depende do tipo de problema e que na geração automática de testes Artificial Bee Algorithm teve o melhor desempenho e o Bat Algorithm é o mais rápido a executar. Nós escolhemos implementar dez algoritmos de inteligência de grupo que possuem várias características diferentes, com diferentes graus de popularidade e que incluem algoritmos antigos e recentes. Os algoritmos escolhidos são: Genetic Bee Colony (GBC) Algorithm, Fish Swarm Algorithm (FSA), Cat Swarm Optimization (CSO), Whale Optimization Algorithm (WOA), Artificial Algae Algorithm (AAA), Elephant Herding Optimization (EHO), Chicken Swarm Optimization Algorithm (CSOA), Moth Flame Optimization (MFO) Algorithm, Grey Wolf Optimization (GWO) Algorithm and Particle Swarm Optimizer (PSO). Para representar os algoritmos evolucionários e servir de comparação contra os algoritmos de inteligência de grupo, escolhemos o Standard Genetic Algorithm (Standard GA), Many-Objective Sorting Algorithm (MOSA) e o Dynamic ManyObjective Sorting Algorithm (DynaMOSA). Este último é o estado da arte atual. Além destes algoritmos, foi implementado mais um algoritmo que é um híbrido (fusão de algoritmos de inteligência de grupo e evolucionários), o Elephant Dynamic Many-Objective Sorting Algorithm (Elephant-DynaMOSA). O EvoSuite foi a ferramenta de geração de testes escolhida para implementar o híbrido e os dez algoritmos de inteligência de grupo por já possuir diversas otimizações, os algoritmos evolucionários já estão implementados e a natureza modular da ferramenta permite facilmente adicionar novos algoritmos ao módulo dos algoritmos. O estudo empírico realizado consiste em duas experiências: a calibração dos parâmetros e a avaliação dos algoritmos. Na primeira experiência, escolhemos vários parâmetros e testámos vários valores destes para cada algoritmo. Foi selecionado um subconjunto de 34 classes e testou-se em 30 seeds diferentes durante 60 segundos para se obter os resultados de cada configuração. De seguida, aplicámos o método estatístico Vargha-Delaney de modo a encontrar a melhor configuração de cada algoritmo. A segunda experiência consistiu em correr a melhor configuração de cada algoritmo em 312 classes com 30 seeds durante 60 segundos. Depois, com o intuito de interpretar os resultados obtidos e conseguir ver qual o melhor algoritmo de inteligência de grupo, se os algoritmos de inteligência de grupo são melhores que os três algoritmos evolucionários e quão boa é a performance do algoritmo híbrido, foram usados os métodos estatísticos de Vargha-Delaney e teste de Friedman. Também se observou a relação entre diversos aspetos dos resultados: a cobertura e o número de gerações, cobertura e a pontuação de mutação, cobertura e diversidade e cobertura e tamanho dos testes. Os nossos resultados revelam que EHO foi o melhor algoritmo de inteligência de grupo e que também superou o Standard GA. Porém, tanto DynaMOSA e MOSA mostram-se superior ao EHO. Em relação ao Elephant-DynaMOSA, que é o híbrido do melhor algoritmo de inteligência de grupo e evolucionário, os resultados foram melhores que o EHO, visto que tem um desempenho semelhante ao MOSA. No final, DynaMOSA foi o algoritmo com maior cobertura média e com os melhores resultados estatísticos nos dois métodos usados. Posteriormente, decidimos discutir outras particularidades dos resultados e propusemos três hipóteses: o melhor algoritmo é superior em todas as classes, qualquer algoritmo consegue atingir pelo menos 50% de cobertura em todas as classes e o desempenho aumenta se o tempo de execução aumentar. A primeira hipótese provou-se falsa visto que houve seis algoritmos estatisticamente melhor que os outros em certas classes: Standard GA, MOSA, DynaMOSA, EHO, Elephant-DynaMOSA e FSA. Isto foi provado ao mostrar-se os valores médios de vários aspetos obtidos nas execuções (número de gerações e testes, tamanho dos testes e cobertura), os resultados do método estatístico Vargha-Delaney e o desempenho de cada algoritmo por critério de cobertura de código. A segunda hipótese também se provou falsa porque 17.5% das classes obtiveram menos de 50% de cobertura independentemente do algoritmo usado. Uma das principais razões é a limitação do EvoSuite como ferramenta de testes, por exemplo não conseguir gerar os inputs necessários para correr a classe. A última hipótese foi a única que se provou ser verdadeira. Para responder a esta hipótese, selecionados a melhor configuração por algoritmo e correu-se 312 classes em uma seed durante uma hora. A cobertura média de todos os algoritmos subiu cerca de 7% e 13 dos 14 algoritmos melhoraram a sua cobertura. Também observámos a evolução dos algoritmos durante a execução e apenas uma minoria dos algoritmos mostrou uma melhoria significativa no desempenho após 60 segundos. Por isso, concluiu-se que apesar da melhoria geral no desempenho, tal melhoria poderá não valer a pena devido ao aumento de recursos necessários com um maior orçamento de tempo. Com isto podemos concluir que apesar do DynaMOSA manter-se como o estado da arte, ele não é o melhor em todas as situações. E que os algoritmos de inteligência de grupo mostraram um certo grau de potencial, principalmente o algoritmo híbrido, Elephant-DynaMOSA. Por isso, nós sugerimos que para trabalho futuro se teste mais algoritmos de inteligência de grupo e algoritmos de múltiplos objetivos, com foco em algoritmos híbridos que combinem os melhores aspetos dos vários algoritmos. Outra iniciativa que pode ser realizada é analisar que algoritmos são melhores para cada critério de cobertura e criar um algoritmo múltiplo capaz de se adaptar e otimizar a procura tendo em conta os critérios de cobertura escolhidos.Search-based software testing is an area of software testing that has shown to be quite successful in generating unit test suites optimized for code coverage. This approach uses meta-heuristic algorithms guided by code coverage criteria (e.g., branch coverage) to generate the tests. When it comes to meta-heuristic algorithms, evolutionary algorithms are the current state-of-the-art, having presented the best results in previous studies. However, are evolutionary algorithms truly the best algorithms in this context? What about bio-inspired algorithms, can they also present good results? Will the current state-of-the-art be replaced with a bio-inspired algorithm? In order to answer these and other questions, we performed an empirical study where we evaluated ten bio-inspired algorithms, three evolutionary algorithms and one hybrid algorithm (a mix of bio-inspired and evolutionary algorithms) on a selection of non-trivial open-source classes. EvoSuite was the test generation tool chosen to implement the ten bio-inspired algorithms and the hybrid since it already has several optimizations and the evolutionary algorithms implemented. Our results show that the Elephant Herding Optimization has the best performance among the bio-inspired algorithms and has surpassed the Standard Genetic Algorithm. However, both the Many-Objective Sorting Algorithm (MOSA) and the Dynamic Many Objective Sorting Algorithm (DynaMOSA) showed superior efficiency compared to all ten bio-inspired algorithms. When it comes to the hybrid algorithm, Elephant Dynamic Many-Objective Sorting Algorithm (Elephant-DynaMOSA), it ended up with a similar performance to MOSA but still worse than the current state-of-the-art, DynaMOSA. We also discussed three hypotheses about the results obtained. Although DynaMOSA remains the state-of-the-art algorithm, it is not the best in all classes. Not only so, but the bio-inspired algorithms showed some potential, especially in the case of the hybrid, Elephant-DynaMOSA. Thus, we suggest future work on hybrid algorithms that fuse the best aspects of several algorithms

A test case generation framework based on UML statechart diagram

Early software fault detection offers more flexibility to correct errors in the early development stages. Unfortunately, existing studies in this domain are not sufficiently comprehensive in describing the major processes of the automated test case generation. Furthermore, the algorithms used for test case generation are not provided or well described. Current studies also hardly address loops and parallel paths issues, and achieved low coverage criteria. Therefore, this study proposes a test case generation framework that generates minimized and prioritized test cases from UML statechart diagram with higher coverage criteria. This study, conducted a review of the previous research to identify the issues and gaps related to test case generation, model-based testing, and coverage criteria. The proposed framework was designed from the gathered information based on the reviews and consists of eight components that represent a comprehensive test case generation processes. They are relation table, relation graph, consistency checking, test path minimization, test path prioritization, path pruning, test path generation, and test case generation. In addition, a prototype to implement the framework was developed. The evaluation of the framework was conducted in three phases: prototyping, comparison with previous studies, and expert review. The results reveal that the most suitable coverage criteria for UML statechart diagram are all-states coverage, all-transitions coverage, alltransition-pairs coverage, and all-loop-free-paths coverage. Furthermore, this study achieves higher coverage criteria in all coverage criteria, except for all-state coverage, when compared with the previous studies. The results of the experts’ review show that the framework is practical, easy to implement due to it is suitability to generate the test cases. The proposed algorithms provide correct results, and the prototype is able to generate test case effectively. Generally, the proposed system is well accepted by experts owing to its usefulness, usability, and accuracy. This study contributes to both theory and practice by providing an early alternative test case generation framework that achieves high coverage and can effectively generate test cases from UML statechart diagrams. This research adds new knowledge to the software testing field, especially for testing processes in the model-based techniques, testing activity, and testing tool support

A review paper: optimal test cases for regression testing using artificial intelligent techniques

The goal of the testing process is to find errors and defects in the software being developed so that they can be fixed and corrected before they are delivered to the customer. Regression testing is an essential quality testing technique during the maintenance phase of the program as it is performed to ensure the integrity of the program after modifications have been made. With the development of the software, the test suite becomes too large to be fully implemented within the given test cost in terms of budget and time. Therefore, the cost of regression testing using different techniques should be reduced, here we dealt many methods such as retest all technique, regression test selection technique (RTS) and test case prioritization technique (TCP). The efficiency of these techniques is evaluated through the use of many metrics such as average percentage of fault detected (APFD), average percentage block coverage (APBC) and average percentage decision coverage (APDC). In this paper we dealt with these different techniques used in test case selection and test case prioritization and the metrics used to evaluate their efficiency by using different techniques of artificial intelligent and describe the best of all