Towards more accurate real time testing

The languages Message Sequence Charts (MSC) [1], System Design Language1 (SDL) [2] and Testing and Test Control Notation Testing2 (TTCN-3) [3] have been developed for the design, modelling and testing of complex software systems. These languages have been developed to complement one another in the software development process. Each of these languages has features for describing, analysing or testing the real time properties of systems. Robust toolsets exist which provide integrated environments for the design, analysis and testing of systems, and it is claimed, for the complete development of real time systems. It was shown in [4] however, that there are fundamental problems with the SDL language and its associated tools for modelling and reasoning about real time systems. In this paper we present the limitations of TTCN-3 and propose recommendations which help minimise the timing inaccuracies that would otherwise occur in using the language directly

Test Case Generation for Mutation-based Testing of Timeliness

AbstractTemporal correctness is crucial for real-time systems. Few methods exist to test temporal correctness and most methods used in practice are ad-hoc. A problem with testing real-time applications is the response-time dependency on the execution order of concurrent tasks. Execution order in turn depends on execution environment properties such as scheduling protocols, use of mutual exclusive resources as well as the point in time when stimuli is injected. Model based mutation testing has previously been proposed to determine the execution orders that need to be verified to increase confidence in timeliness. An effective way to automatically generate such test cases for dynamic real-time systems is still needed. This paper presents a method using heuristic-driven simulation to generate test cases

Uncertainty-wise Test Case Generation and Minimization for Cyber-Physical Systems

Cyber-Physical Systems (CPSs) typically operate in highly indeterminateenvironmental conditions, which require the development of testing methods that must explicitly consider uncertainty in test design, test generation, and test optimization. Towards this direction, we propose a set of uncertainty-wise test case generation and test case minimizationstrategies that rely on test ready models explicitly specifying subjective uncertainty. We propose two test case generation strategies and four test case minimizationstrategies based on the Uncertainty Theory and multi-objectivesearch. These strategies include a novel methodology for designing and introducing indeterminacy sources in the environment during test execution and a novel set of uncertainty-wise test verdicts. We performed an extensive empirical study to select the bestalgorithm out of eight commonly used multi-objective search algorithms, for each of the four minimizationstrategies, with five use cases of two industrial CPS case studies. The minimizedset of test cases obtained with the best algorithm for each minimizationstrategy were executedon the two real CPSs. The results showed that our best test strategy managed to observe 51% more uncertainties due to unknown indeterminate behaviorsof the physical environmentsof the CPSs as compared to the other test strategies. Also, the same test strategy managed to observe 118% more unknown uncertainties as compared to the unique number of known uncertainties.submittedVersio

A CORBA-based system for testing distributed systems

Le test est considéré comme une des étapes du cycle de vie d'un logiciel, et la dernière phase de la méthodologie de création de logiciel (analyse, conception, développement, et test). Dans ce mémoire, nous contribuons à la phase de test. Nous définissons les systèmes repartis et étudions les méthodes et les architectures pour tester un système reparti, à savoir : l'architecture de test centralise, l'architecture de test réparti (ou distant), et l'architecture de test coordonné. Si l'architecture centralisée ne pose pas de problème particulier, l'architecture repartie cause plusieurs problèmes en terme de contrôlabilité et d'observabilité, qui sont des caractéristiques fondamentales du test de conformité. Après une présentation des problèmes de contrôlabilité et d'observabilité, nous proposons une solution a ces deux problèmes, qui consiste a utiliser une architecture de test coordonné. Ensuite, nous proposons et concevons une architecture de test coordonné constituée de trois parties : Ie contrôleur de test. Ie système de test, et l'implementation sous test. Ensuite, nous présentons CORBA (Common Object Request Broker Architecture), qui s'occupe de la communication entre les trois parties de notre architecture de test. Nous présentons une implémentation en Java et CORBA de notre architecture de test. Et enfin, nous illustrons l'application de notre architecture pour Ie test d'une version temporisée du protocole X.25.Abstract: Testing is considered as one of the steps in software life cycle and is the last phase in software creation methodology (Analysis, Design, Development, Testing). In this work, we contribute to testing phase. We define distributed systems, and study methods and architectures to test a distributed system, namely: centralized test architecture, distributed (or remote) test architecture, and coordinated test architecture. If the centralized architecture does not pose any particular problem, the distributed architecture raises several problems in terms of controllability and observability , which are fundamental features of conformance testing. After presenting controllability and observability problems, we propose a solution to these two problems, which consists of using a coordinated test architecture. Then, we propose and design a coordinated test architecture, consisting of three parts: Test Controller, Test System and Implementation Under Test. Then, we introduce CORBA (Common Object Request Broker Architecture), which is responsible for communications between the three parts of our test architecture. Then, we present an implementation in Java and CORBA of our test architecture. And finally, we illustrate the application of our architecture for testing a timed version of the X.25 protocol

Energy-Aware Development and Labeling for Mobile Applications

Today, mobile devices such as smart phones and tablets have become ubiquitous and are used everywhere. Millions of software applications can be purchased and installed on these devices, customizing them to personal interests and needs. However, the frequent use of mobile devices has let a new problem become omnipresent: their limited operation time, due to their limited energy capacities. Although energy consumption can be considered as being a hardware problem, the amount of energy required by today’s mobile devices highly depends on their current workloads, being highly influenced by the software running on them. Thus, although only hardware modules are consuming energy, operating systems, middleware services, and mobile applications highly influence the energy consumption of mobile devices, depending on how efficient they use and control hardware modules. Nevertheless, most of today’s mobile applications totally ignore their influence on the devices’ energy consumption, leading to energy wastes, shorter operation times, and thus, frustrated application users. A major reason for this energy-unawareness is the lack for appropriate tooling for the development of energy-aware mobile applications. As many mobile applications are today behaving energy-unaware and various mobile applications providing similar services exist, mobile application users aim to optimize their devices by installing applications being known as energy-saving or energy-aware; meaning that they consume less energy while providing the same services as their competitors. However, scarce information on the applications’ energy usage is available and, thus, users are forced to install and try many applications manually, before finding the applications fulfilling their personal functional, non-functional, and energy requirements. This thesis addresses the lack of tooling for the development of energy-aware mobile applications and the lack of comparability of mobile applications in terms of energy-awareness with the following two contributions: First, it proposes JouleUnit, an energy profiling and testing framework using unit-tests for the execution of application workloads while profiling their energy consumption in parallel. By extending a well-known testing concept and providing tooling integrated into the development environment Eclipse, JouleUnit requires a low learning curve for the integration into existing development and testing processes. Second, for the comparability of mobile applications in terms of energy efficiency, this thesis proposes an energy benchmarking and labeling service. Mobile applications belonging to the same usage domain are energy-profiled while executing a usage-domain specific benchmark in parallel. Thus, their energy consumption for specific use cases can be evaluated and compared afterwards. To abstract and summarize the profiling results, energy labels are derived that summarize the applications’ energy consumption over all evaluated use cases as a simple energy grade, ranging from A to G. Besides, users can decide how to weigh specific use cases for the computation of energy grades, as it is likely that different users use the same applications differently. The energy labeling service has been implemented for Android applications and evaluated for three different usage domains (being web browsers, email clients, and live wallpapers), showing that different mobile applications indeed differ in their energy consumption for the same services and, thus, their comparison is both possible and sensible. To the best of my knowledge, this is the first approach providing mobile application users comparable energy consumption information on mobile applications without installing and testing them on their own mobile devices

GoTcha: An Interactive Debugger for GoT-Based Distributed Systems

Debugging distributed systems is hard. Most of the techniques that have been developed for debugging such systems use either extensive model checking, or postmortem analysis of logs and traces. Interactive debugging is typically a tool that is only effective in single threaded and single process applications, and is rarely applied to distributed systems. While the live observation of state changes using interactive debuggers is effective, it comes with a host of problems in distributed scenarios. In this paper, we discuss the requirements an interactive debugger for distributed systems should meet, the role the underlying distributed model plays in facilitating the debugger, and the implementation of our interactive debugger: GoTcha. GoTcha is a browser based interactive debugger for distributed systems built on the Global Object Tracker (GoT) programming model. We show how the GoT model facilitates the debugger, and the features that the debugger can offer. We also demonstrate a typical debugging workflow

A testbed for embedded systems

Testing and Debugging are often the most difficult phase of software development. This is especially true of embedded systems which are usually concurrent, have real-time performance and correctness constraints and which execute in the field in an environment which may not permit internal scrutiny of the software behaviour. Although good software engineering practices help, they will never eliminate the need for testing and debugging. This is because failings in the specification and design are often only discovered through testing and understanding these failings and how to correct them comes from debugging. These observations suggest that embedded software should be designed in a way which makes testing and debugging easier and that tools which support these activities are required. Due to the often hostile environment in which the finished embedded system will function, it is necessary to have a platform which allows the software to be developed and tested "in vitro". The Testbed system achieves these goals by providing dynamic modification and process migration facilities for use during development as well as powerful monitoring and background debugging support. These facilities are built on a basic run-time harness supporting an event-driven programming model with a global communication mechanism. This programming model is well suited to the reactive nature of embedded systems. The main research contributions of this work are in the areas of finding deadlock-free, path-optimal routings for networks and of dynamic modification with automated conversion of data which may include pointers