Plug & Test at System Level via Testable TLM Primitives

With the evolution of Electronic System Level (ESL) design methodologies, we are experiencing an extensive use of Transaction-Level Modeling (TLM). TLM is a high-level approach to modeling digital systems where details of the communication among modules are separated from the those of the implementation of functional units. This paper represents a first step toward the automatic insertion of testing capabilities at the transaction level by definition of testable TLM primitives. The use of testable TLM primitives should help designers to easily get testable transaction level descriptions implementing what we call a "Plug & Test" design methodology. The proposed approach is intended to work both with hardware and software implementations. In particular, in this paper we will focus on the design of a testable FIFO communication channel to show how designers are given the freedom of trading-off complexity, testability levels, and cos

A cost model for managing producer and consumer risk in availability demonstration testing.

Evaluation and demonstration of system performance against specified requirements is an essential element of risk reduction during the design, development, and production phases of a product lifecycle. Typical demonstration testing focuses on reliability and maintainability without consideration for availability. A practical reason considers the fact that demonstration testing for availability cannot be performed until very late in the product lifecycle when production representative units become available and system integration is completed. At this point, the requirement to field the system often takes priority over demonstration of availability performance. Without proper validation testing, the system can be fielded with reduced mission readiness and increased lifecycle cost. The need exists for availability demonstration testing (ADT) with emphasis on managing risk while minimizing the cost to the user. Risk management must ensure a test strategy that adequately considers producer and consumer risk objectives. This research proposes a methodology for ADT that provides managers and decision makers an improved ability to distinguish between high and low availability systems. A new availability demonstration test methodology is defined that provides a useful strategy for the consumer to mitigate significant risk without sacrificing the cost of time to field a product or capability. A surface navy electronic system case study supports the practical implementation of this methodology using no more than a simple spreadsheet tool for numerical analysis. Development of this method required three significant components which add to the existing body of knowledge. The first was a comparative performance assessment of existing ADT strategies to understand if any preferences exist. The next component was the development of an approach for ADT design that effectively considers time constraints on the test duration. The third component was the development of a procedure for an ADT design which provides awareness of risk levels in time-constrained ADT, and offers an evaluation of alternatives to select the best sub-optimal test plan. Comparison of the different ADT strategies utilized a simulation model to evaluate runs specified by a five-factor, full-factorial design of experiments. Analysis of variance verified that ADT strategies are significantly different with respect to output responses quality of decision and timeliness. Analysis revealed that the fixed number of failure ADT strategy has the lowest deviation from estimated producer and consumer risk, the measure of quality. The sequential ADT strategy had an average error 3.5 times larger and fixed test time strategies displayed error rates 8.5 to 12.7 larger than the best. The fixed test time strategies had superior performance in timeliness, measured by average test duration. The sequential strategy took 24% longer on average, and the fixed number of failure strategy took 2.5 times longer on average than the best. The research evaluated the application of a time constraint on ADT, and determined an increase in producer and consumer risk levels results when test duration is limited from its optimal value. It also revealed that substitution of a specified time constraint formatted for a specific test strategy produced a pair of dependent relationships between risk levels and the critical test value. These relationships define alternative test plans and could be analyzed in a cost context to compare and select the low cost alternative test plan. This result led to the specification of a support tool to enable a decision maker to understand changes to a and ß resulting from constraint of test duration, and to make decisions based on the true risk exposure. The output of this process is a time-constrained test plan with known producer and consumer risk levels

Exploration versus Exploitation Using Kriging Surrogate Modelling in Electromagnetic Design

This paper discusses the use of kriging surrogate modelling in multiobjective design optimisation in electromagnetics. The importance of achieving appropriate balance between exploration and exploitation is emphasised when searching for the global optimum. It is argued that this approach will yield a procedure to solve time consuming electromagnetic design problems efficiently and will also assist the decision making process to achieve a robust design of practical devices considering tolerances and uncertainties

Finding The Lazy Programmer's Bugs

Traditionally developers and testers created huge numbers of explicit tests, enumerating interesting cases, perhaps biased by what they believe to be the current boundary conditions of the function being tested. Or at least, they were supposed to. A major step forward was the development of property testing. Property testing requires the user to write a few functional properties that are used to generate tests, and requires an external library or tool to create test data for the tests. As such many thousands of tests can be created for a single property. For the purely functional programming language Haskell there are several such libraries; for example QuickCheck [CH00], SmallCheck and Lazy SmallCheck [RNL08]. Unfortunately, property testing still requires the user to write explicit tests. Fortunately, we note there are already many implicit tests present in programs. Developers may throw assertion errors, or the compiler may silently insert runtime exceptions for incomplete pattern matches. We attempt to automate the testing process using these implicit tests. Our contributions are in four main areas: (1) We have developed algorithms to automatically infer appropriate constructors and functions needed to generate test data without requiring additional programmer work or annotations. (2) To combine the constructors and functions into test expressions we take advantage of Haskell's lazy evaluation semantics by applying the techniques of needed narrowing and lazy instantiation to guide generation. (3) We keep the type of test data at its most general, in order to prevent committing too early to monomorphic types that cause needless wasted tests. (4) We have developed novel ways of creating Haskell case expressions to inspect elements inside returned data structures, in order to discover exceptions that may be hidden by laziness, and to make our test data generation algorithm more expressive. In order to validate our claims, we have implemented these techniques in Irulan, a fully automatic tool for generating systematic black-box unit tests for Haskell library code. We have designed Irulan to generate high coverage test suites and detect common programming errors in the process

Functional Testing Approaches for "BIFST-able" tlm_fifo

Evolution of Electronic System Level design methodologies, allows a wider use of Transaction-Level Modeling (TLM). TLM is a high-level approach to modeling digital systems that emphasizes on separating communications among modules from the details of functional units. This paper explores different functional testing approaches for the implementation of Built-in Functional Self Test facilities in the TLM primitive channel tlm_fifo. In particular, it focuses on three different test approaches based on a finite state machine model of tlm_fifo, functional fault models, and march tests respectivel

Intensional and Extensional Semantics of Bounded and Unbounded Nondeterminism

We give extensional and intensional characterizations of nondeterministic functional programs: as structure preserving functions between biorders, and as nondeterministic sequential algorithms on ordered concrete data structures which compute them. A fundamental result establishes that the extensional and intensional representations of non-deterministic programs are equivalent, by showing how to construct a unique sequential algorithm which computes a given monotone and stable function, and describing the conditions on sequential algorithms which correspond to continuity with respect to each order. We illustrate by defining may and must-testing denotational semantics for a sequential functional language with bounded and unbounded choice operators. We prove that these are computationally adequate, despite the non-continuity of the must-testing semantics of unbounded nondeterminism. In the bounded case, we prove that our continuous models are fully abstract with respect to may and must-testing by identifying a simple universal type, which may also form the basis for models of the untyped lambda-calculus. In the unbounded case we observe that our model contains computable functions which are not denoted by terms, by identifying a further "weak continuity" property of the definable elements, and use this to establish that it is not fully abstract