Structural testing techniques for the selective revalidation of software

The research in this thesis addresses the subject of regression testing. Emphasis is placed on developing a technique for selective revalidation which can be used during software maintenance to analyse and retest only those parts of the program affected by changes. In response to proposed program modifications, the technique assists the maintenance programmer in assessing the extent of the program alterations, in selecting a representative set of test cases to rerun, and in identifying any test cases in the test suite which are no longer required because of the program changes. The proposed technique involves the application of code analysis techniques and operations research. Code analysis techniques are described which derive information about the structure of a program and are used to determine the impact of any modifications on the existing program code. Methods adopted from operations research are then used to select an optimal set of regression tests and to identify any redundant test cases. These methods enable software, which has been validated using a variety of structural testing techniques, to be retested. The development of a prototype tool suite, which can be used to realise the technique for selective revalidation, is described. In particular, the interface between the prototype and existing regression testing tools is discussed. Moreover, the effectiveness of the technique is demonstrated by means of a case study and the results are compared with traditional regression testing strategies and other selective revalidation techniques described in this thesis

Autonomous Systems, Robotics, and Computing Systems Capability Roadmap: NRC Dialogue

Contents include the following: Introduction. Process, Mission Drivers, Deliverables, and Interfaces. Autonomy. Crew-Centered and Remote Operations. Integrated Systems Health Management. Autonomous Vehicle Control. Autonomous Process Control. Robotics. Robotics for Solar System Exploration. Robotics for Lunar and Planetary Habitation. Robotics for In-Space Operations. Computing Systems. Conclusion

Search-Based Information Systems Migration: Case Studies on Refactoring Model Transformations

Information systems are built to last for decades; however, the reality suggests otherwise. Companies are often pushed to modernize their systems to reduce costs, meet new policies, improve the security, or to be more competitive. Model-driven engineering (MDE) approaches are used in several successful projects to migrate systems. MDE raises the level of abstraction for complex systems by relying on models as first-class entities. These models are maintained and transformed using model transformations (MT), which are expressed by means of transformation rules to transform models from source to target meta-models. The migration process for information systems may take years for large systems. Thus, many changes are going to be introduced to the transformations to reflect the new business requirements, fix bugs, or to meet the updated metamodels. Therefore, the quality of MT should be continually checked and improved during the evolution process to avoid future technical debts. Most MT programs are written as one large module due to the lack of refactoring/modularization and regression testing tools support. In object-oriented systems, composition and modularization are used to tackle the issues of maintainability and testability. Moreover, refactoring is used to improve the non-functional attributes of the software, making it easier and faster for developers to work and manipulate the code. Thus, we proposed an intelligent computational search approach to automatically modularize MT. Furthermore, we took inspiration from a well-defined quality assessment model for object-oriented design to propose a quality assessment model for MT in particular. The results showed a 45% improvement in the developer’s speed to detect or fix bugs, and developers made 40% less errors when performing a task with the optimized version. Since refactoring operations changes the transformation, it is important to apply regression testing to check their correctness and robustness. Thus, we proposed a multi-objective test case selection technique to find the best trade-off between coverage and computational cost. Results showed a drastic speed-up of the testing process while still showing a good testing performance. The survey with practitioners highlighted the need of such maintenance and evolution framework to improve the quality and efficiency of the existing migration process.Ph.D.College of Engineering & Computer ScienceUniversity of Michigan-Dearbornhttp://deepblue.lib.umich.edu/bitstream/2027.42/149153/1/Bader Alkhazi Final Dissertation.pdfDescription of Bader Alkhazi Final Dissertation.pdf : Restricted to UM users only

First International Conference on Ada (R) Programming Language Applications for the NASA Space Station, volume 1

Topics discussed include: test and verification; environment issues; distributed Ada issues; life cycle issues; Ada in Europe; management/training issues; common Ada interface set; and run time issues

Safety software testing implementation based on standard IEC 61508

In the services industry, the autonomy of a system is strictly related to the system’s safety degree. Safety determines the capabilities and requirements that a system will have. Although safety limits the autonomy of systems, it ensures they are operational, reliable and usable. The thesis consists of an analysis of what functional safety should involve, how safety and testing are related and the application of the safety standard IEC 61508 on an autonomous system. The thesis is performed alongside an internship as a test engineer in KONE; in the critical safety software department. Keywords

Test case prioritization

Regression testing is an expensive software engineering activity intended to provide confidence that modifications to a software system have not introduced faults. Test case prioritization techniques help to reduce regression testing cost by ordering test cases in a way that better achieves testing objectives. In this thesis, we are interested in prioritizing to maximize a test suite's rate of fault detection, measured by a metric, APED, trying to detect regression faults as early as possible during testing. In previous work, several prioritization techniques using low-level code coverage information had been developed. These techniques try to maximize APED over a sequence of software releases, not targeting a particular release. These techniques' effectiveness was empirically evaluated. We present a larger set of prioritization techniques that use information at arbitrary granularity levels and incorporate modification information, targeting prioritization at a particular software release. Our empirical studies show significant improvements in the rate of fault detection over randomly ordered test suites. Previous work on prioritization assumed uniform test costs and fault seventies, which might not be realistic in many practical cases. We present a new cost-cognizant metric, APFD[subscript c], and prioritization techniques, together with approaches for measuring and estimating these costs. Our empirical studies evaluate prioritization in a cost-cognizant environment. Prioritization techniques have been developed independently with little consideration of their similarities. We present a general prioritization framework that allows us to express existing prioritization techniques by a framework algorithm using parameters and specific functions. Previous research assumed that prioritization was always beneficial if it improves the APFD metric. We introduce a prioritization cost-benefit model that more accurately captures relevant cost and benefit factors, and allows practitioners to assess whether it is economical to employ prioritization. Prioritization effectiveness varies across programs, versions, and test suites. We empirically investigate several of these factors on substantial software systems and present a classification-tree-based predictor that can help select the most appropriate prioritization technique in advance. Together, these results improve our understanding of test case prioritization and of the processes by which it is performed

Finding Differences in Privilege Protection and their Origin in Role-Based Access Control Implementations

Les applications Web sont très courantes, et ont des besoins de sécurité. L’un d’eux est le contrôle d’accès. Le contrôle d’accès s’assure que la politique de sécurité est respectée. Cette politique définit l’accès légitime aux données et aux opérations de l’application. Les applications Web utilisent régulièrement le contrôle d’accès à base de rôles (en anglais, « Role-Based Access Control » ou RBAC). Les politiques de sécurité RBAC permettent aux développeurs de définir des rôles et d’assigner des utilisateurs à ces rôles. De plus, l’assignation des privilèges d’accès se fait au niveau des rôles. Les applications Web évoluent durant leur maintenance et des changements du code source peuvent affecter leur sécurité de manière inattendue. Pour éviter que ces changements engendrent des régressions et des vulnérabilités, les développeurs doivent revalider l’implémentation RBAC de leur application. Ces revalidations peuvent exiger des ressources considérables. De plus, la tâche est compliquée par l’éloignement possible entre le changement et son impact sur la sécurité (e.g. dans des procédures ou fichiers différents). Pour s’attaquer à cette problématique, nous proposons des analyses statiques de programmes autour de la protection garantie des privilèges. Nous générons automatiquement des modèles de protection des privilèges. Pour ce faire, nous utilisons l’analyse de flux par traversement de patron (en anglais, « Pattern Traversal Flow Analysis » ou PTFA) à partir du code source de l’application. En comparant les modèles PTFA de différentes versions, nous déterminons les impacts des changements de code sur la protection des privilèges. Nous appelons ces impacts de sécurité des différences de protection garantie (en anglais, « Definite Protection Difference » ou DPD). En plus de trouver les DPD entre deux versions, nous établissons une classification des différences reposant sur la théorie des ensembles.----------ABSTRACT : Web applications are commonplace, and have security needs. One of these is access control. Access control enforces a security policy that allows and restricts access to information and operations. Web applications often use Role-Based Access Control (RBAC) to restrict operations and protect security-sensitive information and resources. RBAC allows developers to assign users to various roles, and assign privileges to the roles. Web applications undergo maintenance and evolution. Their security may be affected by source code changes between releases. Because these changes may impact security in unexpected ways, developers need to revalidate their RBAC implementation to prevent regressions and vulnerabilities. This may be resource-intensive. This task is complicated by the fact that the code change and its security impact may be distant (e.g. in different functions or files). To address this issue, we propose static program analyses of definite privilege protection. We automatically generate privilege protection models from the source code using Pattern Traversal Flow Analysis (PTFA). Using differences between versions and PTFA models, we determine privilege-level security impacts of code changes using definite protection differences (DPDs) and apply a set-theoretic classification to them. We also compute explanatory counter-examples for DPDs in PTFA models. In addition, we shorten them using graph transformations in order to facilitate their understanding. We define protection-impacting changes (PICs), changed code during evolution that impact privilege protection. We do so using graph reachability and differencing of two versions’ PTFA models. We also identify a superset of source code changes that contain root causes of DPDs by reverting these changes. We survey the distribution of DPDs and their classification over 147 release pairs of Word-Press, spanning from 2.0 to 4.5.1. We found that code changes caused no DPDs in 82 (56%) release pairs. The remaining 65 (44%) release pairs are security-affected. For these release pairs, only 0.30% of code is affected by DPDs on average. We also found that the most common change categories are complete gains (� 41%), complete losses (� 18%) and substitution (� 20%)

A Bayesian Framework for Software Regression Testing

Software maintenance reportedly accounts for much of the total cost associated with developing software. These costs occur because modifying software is a highly error-prone task. Changing software to correct faults or add new functionality can cause existing functionality to regress, introducing new faults. To avoid such defects, one can re-test software after modifications, a task commonly known as regression testing. Regression testing typically involves the re-execution of test cases developed for previous versions. Re-running all existing test cases, however, is often costly and sometimes even infeasible due to time and resource constraints. Re-running test cases that do not exercise changed or change-impacted parts of the program carries extra cost and gives no benefit. The research community has thus sought ways to optimize regression testing by lowering the cost of test re-execution while preserving its effectiveness. To this end, researchers have proposed selecting a subset of test cases according to a variety of criteria (test case selection) and reordering test cases for execution to maximize a score function (test case prioritization). This dissertation presents a novel framework for optimizing regression testing activities, based on a probabilistic view of regression testing. The proposed framework is built around predicting the probability that each test case finds faults in the regression testing phase, and optimizing the test suites accordingly. To predict such probabilities, we model regression testing using a Bayesian Network (BN), a powerful probabilistic tool for modeling uncertainty in systems. We build this model using information measured directly from the software system. Our proposed framework builds upon the existing research in this area in many ways. First, our framework incorporates different information extracted from software into one model, which helps reduce uncertainty by using more of the available information, and enables better modeling of the system. Moreover, our framework provides flexibility by enabling a choice of which sources of information to use. Research in software measurement has proven that dealing with different systems requires different techniques and hence requires such flexibility. Using the proposed framework, engineers can customize their regression testing techniques to fit the characteristics of their systems using measurements most appropriate to their environment. We evaluate the performance of our proposed BN-based framework empirically. Although the framework can help both test case selection and prioritization, we propose using it primarily as a prioritization technique. We therefore compare our technique against other prioritization techniques from the literature. Our empirical evaluation examines a variety of objects and fault types. The results show that the proposed framework can outperform other techniques on some cases and performs comparably on the others. In sum, this thesis introduces a novel Bayesian framework for optimizing regression testing and shows that the proposed framework can help testers improve the cost effectiveness of their regression testing tasks