Test Case Selection and Prioritization Using Machine Learning: A Systematic Literature Review

Regression testing is an essential activity to assure that software code changes do not adversely affect existing functionalities. With the wide adoption of Continuous Integration (CI) in software projects, which increases the frequency of running software builds, running all tests can be time-consuming and resource-intensive. To alleviate that problem, Test case Selection and Prioritization (TSP) techniques have been proposed to improve regression testing by selecting and prioritizing test cases in order to provide early feedback to developers. In recent years, researchers have relied on Machine Learning (ML) techniques to achieve effective TSP (ML-based TSP). Such techniques help combine information about test cases, from partial and imperfect sources, into accurate prediction models. This work conducts a systematic literature review focused on ML-based TSP techniques, aiming to perform an in-depth analysis of the state of the art, thus gaining insights regarding future avenues of research. To that end, we analyze 29 primary studies published from 2006 to 2020, which have been identified through a systematic and documented process. This paper addresses five research questions addressing variations in ML-based TSP techniques and feature sets for training and testing ML models, alternative metrics used for evaluating the techniques, the performance of techniques, and the reproducibility of the published studies

Modular product platform design

Modular product platforms, sets of common modules that are shared among a product family, can bring cost savings and enable introduction of multiple product variants quicker than without platforms. This thesis describes the current state of modular platform design and identifies gaps in the current state. The gaps were identified through application of three existing methods and by testing their usability and reliability on engineers and engineering students. Existing platform or modular design methods either are meant for (a) single products, (b) identify only module "cores" leaving the final module boundary definition to the designer, and (c) use only a limited set of evaluation criteria. I introduce a clustering algorithm for common module identification that takes into account possible degrees of commonality. This new algorithm can be applied both at physical and functional domains and at any, and even mixed, levels of hierarchy. Furthermore, the algorithm is not limited to a single measure for commonality analysis. To select the candidate modules for the algorithm, a key discriminator is how difficult the interfaces become. I developed an interface complexity metric based on minimizing redesign in case of a design change. The metric is based on multiple expert interviews during two case studies. The new approach was to look at the interface complexity as described by the material, energy, and information flows flowing through the interface. Finally, I introduce a multi criteria platform scorecard for improved evaluation of modular platforms. It helps a company focus on their strategy and benchmark one's own platform to the competitors'. These tools add to the modular platform development process by filling in the gaps identified. The tools are described in the context of the entire platform design process, and the validity of the methods and applicability to platform design is shown through industrial case studies and examples.reviewe

Using Rule-based Structure to Evaluate Rule-based System Testing Completeness: A Case Study of Loci and Quick Test

Rule-based systems are tested by developing a set of inputs which will produce already known outputs. The problem with this form of testing is that the system code is not considered when generating test cases. This makes software testing completeness difficult to measure. This is important because all the computational models are constructed within the code. Therefore, to show the models of the system are tested, it must be shown that the code is tested. Chem uses the Loci rule-based application framework to build computational fluid dynamics models. These models are tested using the Quick Test suite. The data flow structure built by Loci, along with Quick Test, provided a case study for the research. The test suite was compared against three levels of coverage. The measures indicated that the lowest level of coverage was not achieved. This shows us that structural coverage measures can be utilized to measure rule-based system testing completeness

Automated Assembly Time Prediction Tool Using Predefined Mates From CAD Assemblies

Current Design for Assembly (DFA) methods and tools require extensive amounts and types of user inputs to complete the analysis. Since the methods require extensive amounts and types of inputs, certain issues arise: the analysis can become tedious, time consuming, error prone, and not repeatable. These issues eventually lead to the DFA methods being used as a redesign tool or not being implemented at all. The research presented in this thesis addresses the current DFA limitations and issues by developing and implementing an automated assembly time prediction tool that: extracts explicitly defined connections from SolidWorks assembly models, determines the structural complexity vector of the connections, and inputs the complexity vector into trained artificial neural networks (ANNs) to predict an assembly time. The automated assembly time prediction tool does not require any user inputs other than a mated assembly model. To complete the analysis with the automated tool, the user has to open up the assembly model and click on the developed SW add-in button. Since no additional inputs are required to complete the analysis, the results are completely repeatable when given the same SolidWorks assembly model to evaluate. The results in this thesis show that the developed tool can predict a product\u27s assembly time with as little as 4% error or with as much as +68% error depending on the ANN training set used. Eight different ANN training sets are tested in this thesis, the results show that larger more variable ANN training sets typically predict assembly times with less percent error than smaller less variable ANN training sets. Since the tool extracts mates from assembly models, the sensitivity of the method with respect to different mating styles is also investigated. It is determined that the mating style does have an effect on the predicted assembly time, but this effect is typically within the normal variation ranges of existing DFA methods

On quantifying the value of simulation for training and evaluating robotic agents

Un problème récurrent dans le domaine de la robotique est la difficulté à reproduire les résultats et valider les affirmations faites par les scientifiques. Les expériences conduites en laboratoire donnent fréquemment des résultats propres à l'environnement dans lequel elles ont été effectuées, rendant la tâche de les reproduire et de les valider ardues et coûteuses. Pour cette raison, il est difficile de comparer la performance et la robustesse de différents contrôleurs robotiques. Les environnements substituts à faibles coûts sont populaires, mais introduisent une réduction de performance lorsque l'environnement cible est enfin utilisé. Ce mémoire présente nos travaux sur l'amélioration des références et de la comparaison d'algorithmes (``Benchmarking'') en robotique, notamment dans le domaine de la conduite autonome. Nous présentons une nouvelle platforme, les Autolabs Duckietown, qui permet aux chercheurs d'évaluer des algorithmes de conduite autonome sur des tâches, du matériel et un environnement standardisé à faible coût. La plateforme offre également un environnement virtuel afin d'avoir facilement accès à une quantité illimitée de données annotées. Nous utilisons la plateforme pour analyser les différences entre la simulation et la réalité en ce qui concerne la prédictivité de la simulation ainsi que la qualité des images générées. Nous fournissons deux métriques pour quantifier l'utilité d'une simulation et nous démontrons de quelles façons elles peuvent être utilisées afin d'optimiser un environnement proxy.A common problem in robotics is reproducing results and claims made by researchers. The experiments done in robotics laboratories typically yield results that are specific to a complex setup and difficult or costly to reproduce and validate in other contexts. For this reason, it is arduous to compare the performance and robustness of various robotic controllers. Low-cost reproductions of physical environments are popular but induce a performance reduction when transferred to the target domain. This thesis present the results of our work toward improving benchmarking in robotics, specifically for autonomous driving. We build a new platform, the Duckietown Autolabs, which allow researchers to evaluate autonomous driving algorithms in a standardized framework on low-cost hardware. The platform offers a simulated environment for easy access to annotated data and parallel evaluation of driving solutions in customizable environments. We use the platform to analyze the discrepancy between simulation and reality in the case of predictivity and quality of data generated. We supply two metrics to quantify the usefulness of a simulation and demonstrate how they can be used to optimize the value of a proxy environment