Testbench qualification of SystemC TLM protocols through Mutation Analysis

Transaction-level modeling (TLM) has become the de-facto reference modeling style for system-level design and verification of embedded systems. It allows designers to implement high-level communication protocols for simulations up to 1000x faster than at register-transfer level (RTL). To guarantee interoperability between TLM IP suppliers and users, designers implement the TLM communication protocols by relying on a reference standard, such as the standard OSCI for SystemC TLM. Functional correctness of such protocols as well as their compliance to the reference TLM standard are usually verified through user-defined testbenches, which high-quality and completeness play a key role for an efficient TLM design and verification flow. This article presents a methodology to apply mutation analysis, a technique applied in literature for SW testing, for measuring the testbench quality in verifying TLM protocols. In particular, the methodology aims at (i) qualifying the testbenches by considering both the TLM protocol correctness and their compliance to a defined standard (i.e., OSCI TLM), (ii) optimizing the simulation time during mutation analysis by avoiding mutation redundancies, and (iii) driving the designers in the testbench improvement. Experimental results on benchmarks of different complexity and architectural characteristics are reported to analyze the methodology applicability

A Cross-level Verification Methodology for Digital IPs Augmented with Embedded Timing Monitors

Smart systems implement the leading technology advances in the context of embedded devices. Current design methodologies are not suitable to deal with tightly interacting subsystems of different technological domains, namely analog, digital, discrete and power devices, MEMS and power sources. The interaction effects between the components and between the environment and the system must be modeled and simulated at system level to achieve high performance. Focusing on digital subsystem, additional design constraints have to be considered as a result of the integration of multi-domain subsystems in a single device. The main digital design challenges combined with those emerging from the heterogeneous nature of the whole system directly impact on performance, hence propagation delay, of the digital component. In this paper we propose a design approach to enhance the RTL model of a given digital component for the integration in smart systems, and a methodology to verify the added features at system-level. The design approach consists of ``augmenting'' the RTL model through the automatic insertion of delay sensors, which are capable of detecting and correcting timing failures. The verification methodology consists of an automatic flow of two steps. Firstly the augmented model is abstracted to system-level (i.e., SystemC TLM); secondly mutants, which are code mutations to emulate timing failures, are automatically injected into the abstracted model. Experimental results demonstrate the applicability of the proposed design and verification methodology and the effectiveness of the simulation performance

A Cross-level Verification Methodology for Digital IPs Augmented with Embedded Timing Monitors

Smart systems are characterized by the integration in a single device of multi-domain subsystems of different technological domains, namely, analog, digital, discrete and power devices, MEMS, and power sources. Such challenges, emerging from the heterogeneous nature of the whole system, combined with the traditional challenges of digital design, directly impact on performance and on propagation delay of digital components. This article proposes a design approach to enhance the RTL model of a given digital component for the integration in smart systems with the automatic insertion of delay sensors, which can detect and correct timing failures. The article then proposes a methodology to verify such added features at system level. The augmented model is abstracted to SystemC TLM, which is automatically injected with mutants (i.e., code mutations) to emulate delays and timing failures. The resulting TLM model is finally simulated to identify timing failures and to verify the correctness of the inserted delay monitors. Experimental results demonstrate the applicability of the proposed design and verification methodology, thanks to an efficient sensor-aware abstraction methodology, by applying the flow to three complex case studies

Design and verification of SystemC transaction-level models

Transaction-level modeling allows exploring several SoC design architectures, leading to better performance and easier verification of the final product. In this paper, we present an approach to design and verify SystemC models at the transaction level. We integrate the verification as part of the design flow where we first model both the design and the properties (written in Property Specification language) in Unifed Modeling Language (UML); then, we translate them into an intermediate format modeled with AsmL [language based on Abstract State Machines (ASM)]. The AsmL model is used to generate a finite state machine of the design, including the properties. Checking the correctness of the properties is performed on the fly while generating the state machine. Finally, we translate the verified design to SystemC and map the properties to a set of assertions (as monitors in C#) that can be reused to validate the design at lower levels by simulation. For existing SystemC designs, we propose to translate the code back to AsmL in order to apply the same verification approach. At the SystemC level, we also present a genetic algorithm to enhance the assertions coverage. We will ensure the soundness of our approach by proving the correctness of the SystemC-to-AsmL and AsmL-to-SystemC transformations. We illustrate our approach on two case studies including the PCI bus standard and a master/slave generic architecture from the SystemC library

Complete Model-Based Testing Applied to the Railway Domain

Testing is the most important verification technique to assert the correctness of an embedded system. Model-based testing (MBT) is a popular approach that generates test cases from models automatically. For the verification of safety-critical systems, complete MBT strategies are most promising. Complete testing strategies can guarantee that all errors of a certain kind are revealed by the generated test suite, given that the system-under-test fulfils several hypotheses. This work presents a complete testing strategy which is based on equivalence class abstraction. Using this approach, reactive systems, with a potentially infinite input domain but finitely many internal states, can be abstracted to finite-state machines. This allows for the generation of finite test suites providing completeness. However, for a system-under-test, it is hard to prove the validity of the hypotheses which justify the completeness of the applied testing strategy. Therefore, we experimentally evaluate the fault-detection capabilities of our equivalence class testing strategy in this work. We use a novel mutation-analysis strategy which introduces artificial errors to a SystemC model to mimic typical HW/SW integration errors. We provide experimental results that show the adequacy of our approach considering case studies from the railway domain (i.e., a speed-monitoring function and an interlocking-system controller) and from the automotive domain (i.e., an airbag controller). Furthermore, we present extensions to the equivalence class testing strategy. We show that a combination with randomisation and boundary-value selection is able to significantly increase the probability to detect HW/SW integration errors

A Holistic Approach to Functional Safety for Networked Cyber-Physical Systems

Functional safety is a significant concern in today's networked cyber-physical systems such as connected machines, autonomous vehicles, and intelligent environments. Simulation is a well-known methodology for the assessment of functional safety. Simulation models of networked cyber-physical systems are very heterogeneous relying on digital hardware, analog hardware, and network domains. Current functional safety assessment is mainly focused on digital hardware failures while minor attention is devoted to analog hardware and not at all to the interconnecting network. In this work we believe that in networked cyber-physical systems, the dependability must be verified not only for the nodes in isolation but also by taking into account their interaction through the communication channel. For this reason, this work proposes a holistic methodology for simulation-based safety assessment in which safety mechanisms are tested in a simulation environment reproducing the high-level behavior of digital hardware, analog hardware, and network communication. The methodology relies on three main automatic processes: 1) abstraction of analog models to transform them into system-level descriptions, 2) synthesis of network infrastructures to combine multiple cyber-physical systems, and 3) multi-domain fault injection in digital, analog, and network. Ultimately, the flow produces a homogeneous optimized description written in C++ for fast and reliable simulation which can have many applications. The focus of this thesis is performing extensive fault simulation and evaluating different functional safety metrics, \eg, fault and diagnostic coverage of all the safety mechanisms

Finding False Assurance in Formal Verification of Software Systems

Formal verification plays a crucial role in enhancing the reliability of computing systems by mathematically checking the correctness of a program. Although recent years have witnessed lots of research and applications that optimize the formal verification process, the issue of false assurance persists in certain stages of the formal verification pipeline. The false assurance problem is critical as it can easily undermine months if not years of verification efforts. In this thesis, we first generalized the formal verification process. We then identified and analyzed specific stages susceptible to false assurance. Subsequently, a systematization of knowledge pertaining to the false assurance issues observed at these stages is provided, accompanied by a discussion on the existing defense mechanisms that are currently available. Specifically, we focused on the problem of formal specification incompleteness. We presented FAST in this thesis, which is short for underlineFuzzing-underlineAssisted underlineSpecification underlineTesting. FAST examines the spec for incompleteness issues in an automated way: it first locates spec gaps via mutation testing, i.e., by checking whether a code variant conforms to the original spec. If so, FAST further leverages the test suites to infer whether the gap is introduced by intention or by mistake. Depending on the codebase size, FAST may choose to generate code variants in either an enumerative or evolutionary way. FAST is applied to two open-source codebases that feature formal verification and helps to confirm 13 and 21 blind spots in their spec respectively. This highlights the prevalence of spec incompleteness in real-world applications

Realizability of embedded controllers: from hybrid models to correct implementations

Un controller embedded \ue8 un dispositivo (ovvero, un'opportuna combinazione di componenti hardware e software) che, immerso in un ambiente dinamico, deve reagire alle variazioni ambientali in tempo reale. I controller embedded sono largamente adottati in molti contesti della vita moderna, dall'automotive all'avionica, dall'elettronica di consumo alle attrezzature mediche. La correttezza di tali controller \ue8 indubbiamente cruciale. Per la progettazione e per la verifica di un controller embedded, spesso sorge la necessit\ue0 di modellare un intero sistema che includa sia il controller, sia il suo ambiente circostante. La natura di tale sistema \ue8 ibrido. Esso, infatti, \ue8 ottenuto integrando processi ad eventi discreti (i.e., il controller) e processi a tempo continuo (i.e., l'ambiente). Sistemi di questo tipo sono chiamati cyber-physical (CPS) o sistemi ibridi. Le dinamiche di tali sistemi non possono essere rappresentati efficacemente utilizzando o solo un modello (i.e., rappresentazione) discreto o solo un modello continuo. Diversi tipi di modelli possono sono stati proposti per descrivere i sistemi ibridi. Questi si concentrano su obiettivi diversi: modelli dettagliati sono eccellenti per la simulazione del sistema, ma non sono adatti per la sua verifica; modelli meno dettagliati sono eccellenti per la verifica, ma non sono convenienti per i successivi passi di raffinamento richiesti per la progettazione del sistema, e cos\uec via. Tra tutti questi modelli, gli Automi Ibridi (HA) [8, 77] rappresentano il formalismo pi\uf9 efficace per la simulazione e la verifica di sistemi ibridi. In particolare, un automa ibrido rappresenta i processi ad eventi discreti per mezzo di macchine a stati finiti (FSM), mentre i processi a tempo continuo sono rappresentati mediante variabili "continue" la cui dinamica \ue8 specificata da equazioni differenziali ordinarie (ODE) o loro generalizzazioni (e.g., inclusioni differenziali). Sfortunatamente, a causa della loro particolare semantica, esistono diverse difficolt\ue0 nel raffinare un modello basato su automi ibridi in un modello realizzabile e, di conseguenza, esistono difficolt\ue0 nell'automatizzare il flusso di progettazione di sistemi ibridi a partire da automi ibridi. Gli automi ibridi, infatti, sono considerati dispositivi "perfetti e istantanei". Essi adottano una nozione di tempo e di variabili basata su insiemi "densi" (i.e., l'insieme dei numeri reali). Pertanto, gli automi ibridi possono valutare lo stato (i.e., i valori delle variabili) del sistema in ogni istante, ovvero in ogni infinitesimo di tempo, e con la massima precisione. Inoltre, sono in grado di eseguire computazioni o reagire ad eventi di sincronizzazione in modo istantaneo, andando a cambiare la modalit\ue0 di funzionamento del sistema senza alcun ritardo. Questi aspetti sono convenienti a livello di modellazione, ma nessun dispositivo hardware/software potrebbe implementare correttamente tali comportamenti, indipendentemente dalle sue prestazioni. In altre parole, il controller modellato potrebbe non essere implementabile, ovvero, esso potrebbe non essere realizzabile affatto. Questa tesi affronta questo problema proponendo una metodologia completa e gli strumenti necessari per derivare da modelli basati su automi ibridi, modelli realizzabili e le corrispondenti implementazioni corrette. In un modello realizzabile, il controller analizza lo stato del sistema ad istanti temporali discreti, tipicamente fissati dalla frequenza di clock del processore installato sul dispositivo che implementa il controller. Lo stato del sistema \ue8 dato dai valori delle variabili rilevati dai sensori. Questi valori vengono digitalizzati con precisione finita e propagati al controller che li elabora per decidere se cambiare la modalit\ue0 di funzionamento del sistema. In tal caso, il controller genera segnali che, una volta trasmessi agli attuatori, determineranno il cambiamento della modalit\ue0 di funzionamento del sistema. \uc8 necessario tener presente che i sensori e gli attuatori introducono ritardi che seppur limitati, non possono essere trascurati.An embedded controller is a reactive device (e.g., a suitable combination of hardware and software components) that is embedded in a dynamical environment and has to react to environment changes in real time. Embedded controllers are widely adopted in many contexts of modern life, from automotive to avionics, from consumer electronics to medical equipment. Noticeably, the correctness of such controllers is crucial. When designing and verifying an embedded controller, often the need arises to model the controller and also its surrounding environment. The nature of the obtained system is hybrid because of the inclusion of both discrete-event (i.e., controller) and continuous-time (i.e., environment) processes whose dynamics cannot be characterized faithfully using either a discrete or continuous model only. Systems of this kind are named cyber-physical (CPS) or hybrid systems. Different types of models may be used to describe hybrid systems and they focus on different objectives: detailed models are excellent for simulation but not suitable for verification, high-level models are excellent for verification but not convenient for refinement, and so forth. Among all these models, hybrid automata (HA) [8, 77] have been proposed as a powerful formalism for the design, simulation and verification of hybrid systems. In particular, a hybrid automaton represents discrete-event processes by means of finite state machines (FSM), whereas continuous-time processes are represented by using real-numbered variables whose dynamics is specified by (ordinary) differential equation (ODE) or their generalizations (e.g., differential inclusions). Unfortunately, when the high-level model of the hybrid system is a hybrid automaton, several difficulties should be solved in order to automate the refinement phase in the design flow, because of the classical semantics of hybrid automata. In fact, hybrid automata can be considered perfect and instantaneous devices. They adopt a notion of time and evaluation of continuous variables based on dense sets of values (usually R, i.e., Reals). Thus, they can sample the state (i.e., value assignments on variables) of the hybrid system at any instant in such a dense set R 650. Further, they are capable of instantaneously evaluating guard constraints or reacting to incoming events by performing changes in the operating mode of the hybrid system without any delay. While these aspects are convenient at the modeling level, any model of an embedded controller that relies for its correctness on such precision and instantaneity cannot be implemented by any hardware/software device, no matter how fast it is. In other words, the controller is un-realizable, i.e., un-implementable. This thesis proposes a complete methodology and a framework that allows to derive from hybrid automata proved correct in the hybrid domain, correct realizable models of embedded controllers and the related discrete implementations. In a realizable model, the controller samples the state of the environment at periodic discrete time instants which, typically, are fixed by the clock frequency of the processor implementing the controller. The state of the environment consists of the current values of the relevant variables as observed by the sensors. These values are digitized with finite precision and reported to the controller that may decide to switch the operating mode of the environment. In such a case, the controller generates suitable output signals that, once transmitted to the actuators, will effect the desired change in the operating mode. It is worth noting that the sensors will report the current values of the variables and the actuators will effect changes in the rates of evolution of the variables with bounded delays

Mathematics in Software Reliability and Quality Assurance

This monograph concerns the mathematical aspects of software reliability and quality assurance and consists of 11 technical papers in this emerging area. Included are the latest research results related to formal methods and design, automatic software testing, software verification and validation, coalgebra theory, automata theory, hybrid system and software reliability modeling and assessment