Reducing Software Testing Time with Combinatorial Testing and Test Automation

The development of large software systems is a complex and error prone process. Errors may occur at any stage of software development. These errors, sometimes referred to as bugs, can cause great losses in terms of both time and money if not identified and removed as early as possible. Testing a software product is costly, since it takes much time and need to test many combinations of its functions, integrity, performance etc., which can be called as test cases. The company\u27s goal is to reduce the testing time, so that they can save money and deliver the product much faster to the customer. Testing time can be reduced in two main ways, first by reducing number of test cases and second by automating repeatedly testing areas. This paper will discuss fundamentals of testing such as importance and difference of verification and validation, testing throughout the software development life cycle and testing methods, levels and types. Then it will discuss the possibility of reducing the time spent on testing by reducing number of test cases with combinatorial testing and automating repeatedly tested areas with test automation using Selenium tool. Finally it will also shed some light on a real world test automation project with selenium and two integrated develop environments

Animation and formal verification of real-time reactive systems in an object-oriented environment

Real-time reactive systems are characterized by their continuous interaction with their environment through stimulus-response behavior. The safety-critical nature of their domain and their inherent complexity advocate the use of formal methods in the software development process. TROMLAB development environment supports a process model adequate for dealing with the complexity of reactive systems. The foundation of the TROMLAB environment is the Timed Reactive Object Model (TROM), which combines object-oriented and real-time technologies. Simulation is essential in the behavioral analysis of real-time reactive systems; animation allows a visualization of the simulation process. A rigorous trace analysis of simulation scenarios provides insight into the behavior of the collaborating entities in the configuration. This supports validation of systems designed incrementally and iteratively in the software development life-cycle. Moreover, safety-critical systems need to be verified for adherence to stringent safety and liveness properties. The scope of this thesis is two-fold. We first present an animation tool supporting simulation of reactive systems described in the TROM formalism. We include formal specifications of the functionalities of the simulator in VDM specification language. We then introduce a methodology for formal verification of TROM subsystems. The novelty of the methodology lies in the formal verification approach embedded within an object-oriented framework. The simulator and the verification methodology conform respectively to the operational and logical semantics of TROMs

Validation of highly reliable, real-time knowledge-based systems

Knowledge-based systems have the potential to greatly increase the capabilities of future aircraft and spacecraft and to significantly reduce support manpower needed for the space station and other space missions. However, a credible validation methodology must be developed before knowledge-based systems can be used for life- or mission-critical applications. Experience with conventional software has shown that the use of good software engineering techniques and static analysis tools can greatly reduce the time needed for testing and simulation of a system. Since exhaustive testing is infeasible, reliability must be built into the software during the design and implementation phases. Unfortunately, many of the software engineering techniques and tools used for conventional software are of little use in the development of knowledge-based systems. Therefore, research at Langley is focused on developing a set of guidelines, methods, and prototype validation tools for building highly reliable, knowledge-based systems. The use of a comprehensive methodology for building highly reliable, knowledge-based systems should significantly decrease the time needed for testing and simulation. A proven record of delivering reliable systems at the beginning of the highly visible testing and simulation phases is crucial to the acceptance of knowledge-based systems in critical applications

Design and verification of Guidance, Navigation and Control systems for space applications

In the last decades, systems have strongly increased their complexity in terms of number of functions that can be performed and quantity of relationships between functions and hardware as well as interactions of elements and disciplines concurring to the definition of the system. The growing complexity remarks the importance of defining methods and tools that improve the design, verification and validation of the system process: effectiveness and costs reduction without loss of confidence in the final product are the objectives that have to be pursued. Within the System Engineering context, the modern Model and Simulation based approach seems to be a promising strategy to meet the goals, because it reduces the wasted resources with respect to the traditional methods, saving money and tedious works. Model Based System Engineering (MBSE) starts from the idea that it is possible at any moment to verify, through simulation sessions and according to the phase of the life cycle, the feasibility, the capabilities and the performances of the system. Simulation is used during the engineering process and can be classified from fully numerical (i.e. all the equipment and conditions are reproduced as virtual model) to fully integrated hardware simulation (where the system is represented by real hardware and software modules in their operational environment). Within this range of simulations, a few important stages can be defined: algorithm in the loop (AIL), software in the loop (SIL), controller in the loop (CIL), hardware in the loop (HIL), and hybrid configurations among those. The research activity, in which this thesis is inserted, aims at defining and validating an iterative methodology (based on Model and Simulation approach) in support of engineering teams and devoted to improve the effectiveness of the design and verification of a space system with particular interest in Guidance Navigation and Control (GNC) subsystem. The choice of focusing on GNC derives from the common interest and background of the groups involved in this research program (ASSET at Politecnico di Torino and AvioSpace, an EADS company). Moreover, GNC system is sufficiently complex (demanding both specialist knowledge and system engineer skills) and vital for whatever spacecraft and, last but not least the verification of its behavior is difficult on ground because strong limitations on dynamics and environment reproduction arise. Considering that the verification should be performed along the entire product life cycle, a tool and a facility, a simulator, independent from the complexity level of the test and the stage of the project, is needed. This thesis deals with the design of the simulator, called StarSim, which is the real heart of the proposed methodology. It has been entirely designed and developed from the requirements definition to the software implementation and hardware construction, up to the assembly, integration and verification of the first simulator release. In addition, the development of this technology met the modern standards on software development and project management. StarSim is a unique and self-contained platform: this feature allows to mitigate the risk of incompatibility, misunderstandings and loss of information that may arise using different software, simulation tools and facilities along the various phases. Modularity, flexibility, speed, connectivity, real time operation, fidelity with real world, ease of data management, effectiveness and congruence of the outputs with respect to the inputs are the sought-after features in the StarSim design. For every iteration of the methodology, StarSim guarantees the possibility to verify the behavior of the system under test thanks to the permanent availability of virtual models, that substitute all those elements not yet available and all the non-reproducible dynamics and environmental conditions. StarSim provides a furnished and user friendly database of models and interfaces that cover different levels of detail and fidelity, and supports the updating of the database allowing the user to create custom models (following few, simple rules). Progressively, pieces of the on board software and hardware can be introduced without stopping the process of design and verification, avoiding delays and loss of resources. StarSim has been used for the first time with the CubeSats belonging to the e-st@r program. It is an educational project carried out by students and researchers of the “CubeSat Team Polito” in which StarSim has been mainly used for the payload development, an Active Attitude Determination and Control System, but StarSim’s capabilities have also been updated to evaluate functionalities, operations and performances of the entire satellite. AIL, SIL, CIL, HIL simulations have been performed along all the phases of the project, successfully verifying a great number of functional and operational requirements. In particular, attitude determination algorithms, control laws, modes of operation have been selected and verified; software has been developed step by step and the bugs-free executable files have been loaded on the micro-controller. All the interfaces and protocols as well as data and commands handling have been verified. Actuators, logic and electrical circuits have been designed, built and tested and sensors calibration has been performed. Problems such as real time and synchronization have been solved and a complete hardware in the loop simulation test campaign both for A-ADCS standalone and for the entire satellite has been performed, verifying the satisfaction of a great number of CubeSat functional and operational requirements. The case study represents the first validation of the methodology with the first release of StarSim. It has been proven that the methodology is effective in demonstrating that improving the design and verification activities is a key point to increase the confidence level in the success of a space mission

An Adaptive Design Methodology for Reduction of Product Development Risk

Embedded systems interaction with environment inherently complicates understanding of requirements and their correct implementation. However, product uncertainty is highest during early stages of development. Design verification is an essential step in the development of any system, especially for Embedded System. This paper introduces a novel adaptive design methodology, which incorporates step-wise prototyping and verification. With each adaptive step product-realization level is enhanced while decreasing the level of product uncertainty, thereby reducing the overall costs. The back-bone of this frame-work is the development of Domain Specific Operational (DOP) Model and the associated Verification Instrumentation for Test and Evaluation, developed based on the DOP model. Together they generate functionally valid test-sequence for carrying out prototype evaluation. With the help of a case study 'Multimode Detection Subsystem' the application of this method is sketched. The design methodologies can be compared by defining and computing a generic performance criterion like Average design-cycle Risk. For the case study, by computing Average design-cycle Risk, it is shown that the adaptive method reduces the product development risk for a small increase in the total design cycle time.Comment: 21 pages, 9 figure