Software component testing : a standard and the effectiveness of techniques

This portfolio comprises two projects linked by the theme of software component testing, which is also often referred to as module or unit testing. One project covers its standardisation, while the other considers the analysis and evaluation of the application of selected testing techniques to an existing avionics system. The evaluation is based on empirical data obtained from fault reports relating to the avionics system. The standardisation project is based on the development of the BC BSI Software Component Testing Standard and the BCS/BSI Glossary of terms used in software testing, which are both included in the portfolio. The papers included for this project consider both those issues concerned with the adopted development process and the resolution of technical matters concerning the definition of the testing techniques and their associated measures. The test effectiveness project documents a retrospective analysis of an operational avionics system to determine the relative effectiveness of several software component testing techniques. The methodology differs from that used in other test effectiveness experiments in that it considers every possible set of inputs that are required to satisfy a testing technique rather than arbitrarily chosen values from within this set. The three papers present the experimental methodology used, intermediate results from a failure analysis of the studied system, and the test effectiveness results for ten testing techniques, definitions for which were taken from the BCS BSI Software Component Testing Standard. The creation of the two standards has filled a gap in both the national and international software testing standards arenas. Their production required an in-depth knowledge of software component testing techniques, the identification and use of a development process, and the negotiation of the standardisation process at a national level. The knowledge gained during this process has been disseminated by the author in the papers included as part of this portfolio. The investigation of test effectiveness has introduced a new methodology for determining the test effectiveness of software component testing techniques by means of a retrospective analysis and so provided a new set of data that can be added to the body of empirical data on software component testing effectiveness

Android-Based Claim System for Electricity Network Customers of PLN Padang Branch

The Padang branch of the State Electricity Company (PLN) is a BUMN responsible for the electricity aspect. Complaints about various electricity problems are difficult for customers, so it is necessary to have a system that can be used as a place for protests and increase PLN's loyalty to customers. The Android-based Claim System is a new application developed to be used by customers as a forum for complaints about electrical problems to the Padang branch of PLN. The Claim System is made into two parts: a Website-based System for PLN and an Android-based Claim Application for customers. Application development using the OOP concept uses UML (Unified Modeling Language) diagrams with the PHP MySql and Android programming languages. The test results were carried out using black box testing with the relevant results. The data of 35 respondents from the assessment questionnaire on the claims system obtained results with an Excellent rating of 78%, a Good rating of 21%, and a Bad rating of 2%. From the analysis results, the Claim System supported by this Android-based Client application can help customers complain about electrical problems quickly and easily.The Padang branch of the State Electricity Company (PLN) is a BUMN responsible for the electricity aspect. Complaints about various electricity problems are difficult for customers, so it is necessary to have a system that can be used as a place for protests and increase PLN's loyalty to customers. The Android-based Claim System is a new application developed to be used by customers as a forum for complaints about electrical problems to the Padang branch of PLN. The Claim System is made into two parts: a Website-based System for PLN and an Android-based Claim Application for customers. Application development using the OOP concept uses UML (Unified Modeling Language) diagrams with the PHP MySql and Android programming languages. The test results were carried out using black box testing with the relevant results. The data of 35 respondents from the assessment questionnaire on the claims system obtained results with an Excellent rating of 78%, a Good rating of 21%, and a Bad rating of 2%. From the analysis results, the Claim System supported by this Android-based Client application can help customers complain about electrical problems quickly and easily

JWalk: a tool for lazy, systematic testing of java classes by design introspection and user interaction

Popular software testing tools, such as JUnit, allow frequent retesting of modified code; yet the manually created test scripts are often seriously incomplete. A unit-testing tool called JWalk has therefore been developed to address the need for systematic unit testing within the context of agile methods. The tool operates directly on the compiled code for Java classes and uses a new lazy method for inducing the changing design of a class on the fly. This is achieved partly through introspection, using Java’s reflection capability, and partly through interaction with the user, constructing and saving test oracles on the fly. Predictive rules reduce the number of oracle values that must be confirmed by the tester. Without human intervention, JWalk performs bounded exhaustive exploration of the class’s method protocols and may be directed to explore the space of algebraic constructions, or the intended design state-space of the tested class. With some human interaction, JWalk performs up to the equivalent of fully automated state-based testing, from a specification that was acquired incrementally

Formale Verifikationsmethodiken für nichtlineare analoge Schaltungen

The objective of this thesis is to develop new methodologies for formal verification of nonlinear analog circuits. Therefore, new approaches to discrete modeling of analog circuits, specification of analog circuit properties and formal verification algorithms are introduced. Formal approaches to verification of analog circuits are not yet introduced into industrial design flows and still subject to research. Formal verification proves specification conformance for all possible input conditions and all possible internal states of a circuit. Automatically proving that a model of the circuit satisfies a declarative machine-readable property specification is referred to as model checking. Equivalence checking proves the equivalence of two circuit implementations. Starting from the state of the art in modeling analog circuits for simulation-based verification, discrete modeling of analog circuits for state space-based formal verification methodologies is motivated in this thesis. In order to improve the discrete modeling of analog circuits, a new trajectory-directed partitioning algorithm was developed in the scope of this thesis. This new approach determines the partitioning of the state space parallel or orthogonal to the trajectories of the state space dynamics. Therewith, a high accuracy of the successor relation is achieved in combination with a lower number of states necessary for a discrete model of equal accuracy compared to the state-of-the-art hyperbox-approach. The mapping of the partitioning to a discrete analog transition structure (DATS) enables the application of formal verification algorithms. By analyzing digital specification concepts and the existing approaches to analog property specification, the requirements for a new specification language for analog properties have been discussed in this thesis. On the one hand, it shall meet the requirements for formal specification of verification approaches applied to DATS models. On the other hand, the language syntax shall be oriented on natural language phrases. By synthesis of these requirements, the analog specification language (ASL) was developed in the scope of this thesis. The verification algorithms for model checking, that were developed in combination with ASL for application to DATS models generated with the new trajectory-directed approach, offer a significant enhancement compared to the state of the art. In order to prepare a transition of signal-based to state space-based verification methodologies, an approach to transfer transient simulation results from non-formal test bench simulation flows into a partial state space representation in form of a DATS has been developed in the scope of this thesis. As has been demonstrated by examples, the same ASL specification that was developed for formal model checking on complete discrete models could be evaluated without modifications on transient simulation waveforms. An approach to counterexample generation for the formal ASL model checking methodology offers to generate transition sequences from a defined starting state to a specification-violating state for inspection in transient simulation environments. Based on this counterexample generation, a new formal verification methodology using complete state space-covering input stimuli was developed. By conducting a transient simulation with these complete state space-covering input stimuli, the circuit adopts every state and transition that were visited during stimulus generation. An alternative formal verification methodology is given by retransferring the transient simulation responses to a DATS model and by applying the ASL verification algorithms in combination with an ASL property specification. Moreover, the complete state space-covering input stimuli can be applied to develop a formal equivalence checking methodology. Therewith, the equivalence of two implementations can be proven for every inner state of both systems by comparing the transient simulation responses to the complete-coverage stimuli of both circuits. In order to visually inspect the results of the newly introduced verification methodologies, an approach to dynamic state space visualization using multi-parallel particle simulation was developed. Due to the particles being randomly distributed over the complete state space and moving corresponding to the state space dynamics, another perspective to the system's behavior is provided that covers the state space and hence offers formal results. The prototypic implementations of the formal verification methodologies developed in the scope of this thesis have been applied to several example circuits. The acquired results for the new approaches to discrete modeling, specification and verification algorithms all demonstrate the capability of the new verification methodologies to be applied to complex circuit blocks and their properties.Gegenstand dieser Dissertation ist die Entwicklung neuer Methodiken zur formalen Verifikation nichtlinearer analoger elektronischer Schaltungen. Dazu werden im Rahmen dieser Arbeit entstandene neue Ansätze in den Bereichen verifikationsgerechte diskrete Modellierung analoger Schaltungen, Spezifikation analoger Schaltungseigenschaften und formale Verifikationsalgorithmen vorgestellt. Ausgehend vom Stand der Technik der Modellierung analoger Schaltungen für die simulationsbasierte Verifikation wird im Rahmen dieser Arbeit die diskrete Modellierung analoger Schaltungen für zustandsraumbasierte formale Verifikationsverfahren betrachtet. Dazu wurde ein neuer Ansatz zur diskreten Modellierung entwickelt, der die Aufteilungsstruktur anhand der Trajektorien der Vektorfelddynamik bestimmt. So wird eine hohe Genauigkeit der Nachfolgerrelation ermöglicht, woraus eine niedrigere Zahl an Zuständen für ein diskretes Modell gleicher Genauigkeit im Vergleich mit dem bisherigen Stand der Technik folgt. Die Abbildung der Trajektorien-gesteuerten Partitionierung auf eine diskrete analoge Transitionsstruktur (DATS) erlaubt die Anwendung von formalen Verifikationsalgorithmen. Die formale Spezifikation von Eigenschaften in ersten Ansätzen zum Model Checking analoger Schaltungen hat sich stark an den bestehenden temporallogischen Verfahren aus dem Bereich digitaler Hardware orientiert. Ausgehend von einer Analyse digitaler Spezifikationskonzepte und der bestehenden Ansätze für analoge Eigenschaften wurden Anforderungen an eine neue Spezifikationssprache in dieser Arbeit abgeleitet. Die aus diesen Anforderungen im Rahmen dieser Arbeit entwickelte analoge Spezifikationssprache "Analog Specification Language" (ASL) basiert auf einer natürlichsprachlichen Kapselung temporallogischer Operationen, die mit erweiterten Algorithmen zur Transitionspfadbestimmung, Durchführung von Berechnungen auf Zustandsparametern und Oszillationsbestimmung eine hohe Ausdrucksstärke analoger Eigenschaften mit einer anwenderfreundlichen Syntax kombinieren konnte. Die zusammen mit ASL entwickelten Model Checking-Verifikationsalgorithmen zur Auswertung von ASL-Spezifikationen auf einem mit dem Trajektorien-gesteuerten Diskretisierungsverfahren erzeugten DATS-Modell bilden eine wesentliche Erweiterung zum Stand der Technik. Um einen Übergang der Verifikation von signalbasierten zu zustandsraumbasierten Methodiken zu ermöglichen, wurde im Rahmen dieser Arbeit ein Ansatz entwickelt, der die Übertragung von transienten Simulationsergebnissen aus nicht-formalen Testbench-Simulationsumgebungen in eine partielle DATS-Zustandsraumdarstellung ermöglicht. Damit kann, wie anhand von Beispielen gezeigt werden konnte, die gleiche ASL-Spezifikation für Eigenschaften eines vollständigen diskreten Modells ohne Modifikation auch auf Simulationsergebnissen ausgewertet werden. Ein für das formale ASL-basierte Model Checking entwickelter Ansatz zur Erzeugung von Gegenbeispielen für als spezifikationsverletzend identifizierte Zustandsraumgebiete erlaubt es, Transitionsfolgen von einem definierten Startzustand zu einem spezifikationsverletzenden Zustand zu ermitteln. Auf Basis dieses Gegenbeispiel-Verfahrens wurde eine neue formale Eigenschaftsverifikationsmethodik mittels vollständig den Zustandsraum einer Schaltung abdeckenden Eingangsstimuli entwickelt. Die vollständig den Zustandsraum abdeckenden Eingangsstimuli bieten noch eine weitere Anwendungsmöglichkeit im Bereich des Äquivalenzvergleichs. Die im Rahmen dieser Arbeit entwickelte Methodik zum formalen Äquivalenzvergleich auf Basis der vollständig den Zustandsraum abdeckenden Eingangsstimuli ersetzt die anwenderdefinierten Eingangsstimuli durch die vollständig den Zustandsraum abdeckenden. So kann die Äquivalenz für jeden möglichen Zustand der zu vergleichenden Implementierungen anhand eines automatisierten Vergleichs der Simulationsergebnisse beider Implementierungen gezeigt werden. Um die Ergebnisse der neu eingeführten formalen Verifikationsmethodiken visuell zu untersuchen wurde ein Verfahren entwickelt, das den Zustandsraum und seine Dynamik mittels eines Partikel-Simulationsansatzes visualisiert. Da die Partikel über den gesamten Zustandsraum randomisiert verteilt werden und sich dann gemäß der Vektorfelddynamik fortbewegen, kann auch hier ein Einblick in das Systemverhalten gewonnen werden, der eine weitestgehend vollständige und somit formale Repräsentation des Zustandsraums bietet. Die prototypische Implementierung der im Rahmen dieser Arbeit entwickelten formalen Verifikationsmethodiken wurde auf zahlreiche Beispielschaltungen angewendet. Die Ergebnisse für die neuen Ansätze zur diskreten Modellierung, zur Spezifikation und zu Verifikationsalgorithmen analoger Schaltungen zeigen, dass die aus diesen Ansätzen erzeugten Verifikationsmethodiken erfolgreich auf komplexe Zustandsraumstrukturen angewendet werden können

Automated Web Applications Testing

Unit tests are a vital part of several software development practices and processes such as Test-First Programming, Extreme Programming and Test-Driven Development. This article shortly presents the software quality and testing concepts as well as an introduction to an automated unit testing framework for PHP web based applicationssoftware quality, continuous integration, unit testing

Formal Verification throughout the Development of Robust Systems

As transistors are becomming smaller and smaller, they become more susceptible to transient faults due to radiation. A system can be modified to handle these faults and prevent errors that are visible from outside. We present a formal method for equivalence checking to verify that this modification does not change the nominal behavior of the system. On the other hand, we contribute an algorithm to formally verify that a circuit is robust against transient faults under all possible input assignments and variability. If equivalence or robustness cannot be shown, a counterexample is generated

The construction of oracles for software testing

Software testing is important throughout the software life cycle. Testing is the part of the software development process where a computer program is subject to specific conditions to show that the problem meets its intended design. Building a testing oracle is one part of software testing. An oracle is an external mechanism which can be used to check test output for correctness. The characteristics of available oracles have a dominating influence on the cost and quality of software testing. In this thesis, methods of constructing oracles are investigated and classified. There are three kinds of method of constructing oracles: the pseudo-oracle approach, oracles using attributed grammars and oracles based on formal specification. This thesis develops a method for constructing an oracle, based on the Z specification language. A specification language can describe the correct syntax and semantics of software. The contextual part of a specification describes all the legal input to the program and the semantics part describes the meaning of the given input data. Based on this idea, an oracle is constructed and a prototype is implemented according to the method proposed in the thesis