Model based testing for agent systems

Although agent technology is gaining world wide popularity, a hindrance to its uptake is the lack of proper testing mechanisms for agent based systems. While many traditional software testing methods can be generalized to agent systems, there are many aspects that are different and which require an understanding of the underlying agent paradigm. In this paper we present certain aspects of a testing framework that we have developed for agent based systems. The testing framework is a model based approach using the design models of the Prometheus agent development methodology. In this paper we focus on unit testing and identify the appropriate units, present mechanisms for generating suitable test cases and for determining the order in which the units are to be tested, present a brief overview of the unit testing process and an example. Although we use the design artefacts from Prometheus the approach is suitable for any plan and event based agent system

Complete Agent-driven Model-based System Testing for Autonomous Systems

In this position paper, a novel approach to testing complex autonomous transportation systems (ATS) in the automotive, avionic, and railway domains is described. It is intended to mitigate some of the most critical problems regarding verification and validation (V&V) effort for ATS. V&V is known to become infeasible for complex ATS, when using conventional methods only. The approach advocated here uses complete testing methods on the module level, because these establish formal proofs for the logical correctness of the software. Having established logical correctness, system-level tests are performed in simulated cloud environments and on the target system. To give evidence that 'sufficiently many' system tests have been performed with the target system, a formally justified coverage criterion is introduced. To optimise the execution of very large system test suites, we advocate an online testing approach where multiple tests are executed in parallel, and test steps are identified on-the-fly. The coordination and optimisation of these executions is achieved by an agent-based approach. Each aspect of the testing approach advocated here is shown to either be consistent with existing standards for development and V&V of safety-critical transportation systems, or it is justified why it should become acceptable in future revisions of the applicable standards.Comment: In Proceedings FMAS 2021, arXiv:2110.1152

Towards the Development of a Simulator for Investigating the Impact of People Management Practices on Retail Performance

Often models for understanding the impact of management practices on retail performance are developed under the assumption of stability, equilibrium and linearity, whereas retail operations are considered in reality to be dynamic, non-linear and complex. Alternatively, discrete event and agent-based modelling are approaches that allow the development of simulation models of heterogeneous non-equilibrium systems for testing out different scenarios. When developing simulation models one has to abstract and simplify from the real world, which means that one has to try and capture the 'essence' of the system required for developing a representation of the mechanisms that drive the progression in the real system. Simulation models can be developed at different levels of abstraction. To know the appropriate level of abstraction for a specific application is often more of an art than a science. We have developed a retail branch simulation model to investigate which level of model accuracy is required for such a model to obtain meaningful results for practitioners.Comment: 24 pages, 7 figures, 6 tables, Journal of Simulation 201

Decision Support for Negotiations among Microgrids Using a Multiagent Architecture

[EN] This paper presents a decision support model for negotiation portfolio optimization considering the participation of players in local markets (at the microgrid level) and in external markets, namely in regional markets, wholesale negotiations and negotiations of bilateral agreements. A local internal market model for microgrids is defined, and the connection between interconnected microgrids is based on nodal pricing to enable negotiations between nearby microgrids. The market environment considering the local market setting and the interaction between integrated microgrids is modeled using a multi-agent approach. Several multi-agent systems are used to model the electricity market environment, the interaction between small players at a microgrid scale, and to accommodate the decision support features. The integration of the proposed models in this multi-agent society and interaction between these distinct specific multi-agent systems enables modeling the system as a whole and thus testing and validating the impact of the method in the outcomes of the involved players. Results show that considering the several negotiation opportunities as complementary and making use of the most appropriate markets depending on the expected prices at each moment allows players to achieve more profitable results

A Framework for Model-based Testing of Integrated Modular Avionics

In modern aircraft, electronics and control systems are designed based on the Integrated Modular Avionics (IMA) system architecture. While this has numerous advantages (reduction of weight, reduced power and fuel consumption, reduction of development cost and certification effort), the IMA platform also adds an additional layer of complexity. Due to the safety-critical nature of many avionics functions careful and accurate verification and testing are imperative. This thesis describes results achieved from research on model-based testing of IMA systems, in part obtained during the European research project SCARLETT. It presents a complete framework which enables IMA domain experts to design and run model-based tests on bare module, configured module, and application level in a standardised test environment. The first part of this thesis provides background information on the relevant topics: the IMA concept, domain-specific languages, model-based testing, and the TTCN-3 standard. The second part introduces the IMA Test Modelling Language (ITML) framework and its components. It describes a tailored TTCN-3 test environment with appropriate adapters and codecs. Based on MetaEdit and its meta-metamodel GOPPRR, it defines the three variants of the domain-specific language ITML, each with its abstract and concrete syntax as well as static and dynamic semantics. The process of test procedure generation from ITML models is explained in detail. Furthermore, the design and implementation of a universal Test Agent is shown. A dedicated communication protocol for controlling the agent is defined as well. The third part provides an evaluation of the framework. It shows usage scenarios in the SCARLETT project, gives a comparison to related tools and approaches, and explains the advantages of using the ITML framework for an IMA domain expert. The final part presents several example ITML models. It also provides reference material like XML schemata, framework source code, and model validators

An Expressive Language and Efficient Execution System for Software Agents

Software agents can be used to automate many of the tedious, time-consuming information processing tasks that humans currently have to complete manually. However, to do so, agent plans must be capable of representing the myriad of actions and control flows required to perform those tasks. In addition, since these tasks can require integrating multiple sources of remote information ? typically, a slow, I/O-bound process ? it is desirable to make execution as efficient as possible. To address both of these needs, we present a flexible software agent plan language and a highly parallel execution system that enable the efficient execution of expressive agent plans. The plan language allows complex tasks to be more easily expressed by providing a variety of operators for flexibly processing the data as well as supporting subplans (for modularity) and recursion (for indeterminate looping). The executor is based on a streaming dataflow model of execution to maximize the amount of operator and data parallelism possible at runtime. We have implemented both the language and executor in a system called THESEUS. Our results from testing THESEUS show that streaming dataflow execution can yield significant speedups over both traditional serial (von Neumann) as well as non-streaming dataflow-style execution that existing software and robot agent execution systems currently support. In addition, we show how plans written in the language we present can represent certain types of subtasks that cannot be accomplished using the languages supported by network query engines. Finally, we demonstrate that the increased expressivity of our plan language does not hamper performance; specifically, we show how data can be integrated from multiple remote sources just as efficiently using our architecture as is possible with a state-of-the-art streaming-dataflow network query engine

Variational Exploration Module VEM: A Cloud-Native Optimization and Validation Tool for Geospatial Modeling and AI Workflows

Geospatial observations combined with computational models have become key to understanding the physical systems of our environment and enable the design of best practices to reduce societal harm. Cloud-based deployments help to scale up these modeling and AI workflows. Yet, for practitioners to make robust conclusions, model tuning and testing is crucial, a resource intensive process which involves the variation of model input variables. We have developed the Variational Exploration Module which facilitates the optimization and validation of modeling workflows deployed in the cloud by orchestrating workflow executions and using Bayesian and machine learning-based methods to analyze model behavior. User configurations allow the combination of diverse sampling strategies in multi-agent environments. The flexibility and robustness of the model-agnostic module is demonstrated using real-world applications.Comment: Submitted to IAAI 2024: Deployed Innovative Tools for Enabling AI Application

Automated testing for intelligent agent systems

This paper describes an approach to unit testing of plan based agent systems, with a focus on automated generation and execution of test cases. Design artefacts, supplemented with some additional data, provide the basis for specification of a comprehensive suite of test cases. Correctness of execution is evaluated against a design model, and a comprehensive report of errors and warnings is provided to the user. Given that it is impossible to design test suites which execute all possible traces of an agent program, it is extremely important to thoroughly test all units in as wide a variety of situations as possible to ensure acceptable behaviour. We provide details of the information required in design models or related data to enable the automated generation and execution of test cases. We also briefly describe the implemented tool which realises this approach