10,858 research outputs found
Simulation-based testing of highly configurable cyber-physical systems: automation, optimization and debugging
Sistema Ziber-Fisikoek sistema ziber digitalak sistema fisikoekin uztartzen dituzte. Sistema hauen aldakortasuna handitzen ari da erabiltzaileen hainbat behar betetzeko. Ondorioz, sistema ziber-fisikoa aldakorrak edota produktu lerroak ari dira garatzen eta sistema hauek milaka edo milioika konfiguraziotan konfiguratu daitezke. Sistema ziber-fisiko aldakorren test eta balidazioa prozesua garestia da, batez ere probatu beharreko konfigurazio kopuruaren ondorioz. Konfigurazio kopuru altuak sistemaren prototipo bat erabiltzea ezinezkoa egiten du. Horregatik, sistema ziber-fisiko aldagarriak simulazio modeloak erabilita probatzen dira. Hala ere, simulazio bidez sistema
ziber-fisikoak probatzea erronka izaten jarraitzen du. Hasteko, simulazio denbora altua izaten da normalki, software-az aparte, sistema fisikoa simulatu behar delako. Sistema fisiko hau normalean modelo matematiko konplexuen bitartez modelatzen da, konputazionalki garestia delarik. Jarraitzeko, sistema ziber-fisikoek ingeniaritzaren domeinu ezberdinak dituzte tartean, adibidez mekanika edo elektronika. Domeinu bakoitzak bere simulazio erremienta erabiltzen du, eta erremienta guzti hauek interkonektatzeko ko-simulazioa erabiltzen da. Nahiz eta ko-simulazioa abantaila bat izan ematen duen flexibilitateagatik, simulagailu ezberdinen erabilerak simulazio denbora handiagotzen du. Azkenik, sistema ziber-fisikoak simulaziopean probatzean, probak maila ezberdinetan egin behar dira (adb., Model, Software eta Hardware-in-the-Loop mailak), eta honek, proba-kasuak exekutatzeko denbora handitzen du. Tesi honen helburua sistema ziber-fisiko aldakorren test jardunbideak hobetzea da, horretarako automatizazio, optimizazio eta arazketa metodoak proposatzen ditu. Automatizazioari dagokionez, lehenengo, erremienta-bidezko metodologia bat proposatzen da. Metodologia hau test sistema instantziak automatikoki sortzeko gai
da, test sistema hauek sistema ziber-fisiko aldagarrien konfigurazioak automatikoki probatzeko gai dira (adb., test orakuluen bitartez). Bigarren, test frogak automatikoki sortzeko planteamendu bat proposatzen da helburu anitzeko bilaketa algoritmoak erabilita. Optimizazioari dagokionez, test frogen aukeraketarako planteamendu bat eta test frogen priorizaziorako beste planteamendu bat proposatzen dira, biak bilaketa alix goritmoak erabiliz, sistema ziber-fisiko aldakorrak test maila ezberdinetan probatzeko helburuarekin. Arazketari dagokionez, “espektroan oinarritutako falten lokalizazioa” izeneko teknika bat produktu lerroen testuingurura adaptatu da, eta faltak isolatzeko metodo bat proposatzen da. Honek, falta ezberdinak lokalizatzea errezten du ez bakarrik sistema ziber-fisiko aldakorretan, baizik eta edozein produktu lerrotan non “feature model” delako modeloak erabiltzen diren aldakortasuna kudeatzeko.Los sistemas cyber-físicos (CPSs) integran tecnologías digitales con procesos físicos. La variabilidad de estos sistemas está creciendo para responder a la demanda de diferentes clientes. Como consecuencia de ello, los CPSs están volviéndose configurables e incluso líneas de producto, lo que significa que pueden ser configurados en miles y millones de configuraciones. El testeo de sistemas cyber-físicos configurables es un proceso costoso, en general debido a la cantidad de configuraciones que han de ser testeadas. El número de configuraciones a testear hace imposible el uso de un prototipo del sistema. Por ello, los sistemas CPSs configurables están siendo testeadas utilizando modelos de simulación. Sin embargo, el testeo de sistemas cyber-físicos
bajo simulación sigue siendo un reto. Primero, el tiempo de simulación es normalmente largo, ya que, además del software, la capa física del CPS ha de ser testeada. Esta capa física es típicamente modelada con modelos matemáticos complejos, lo cual es computacionalmente caro. Segundo, los sistemas cyber-físicos implican el uso de diferentes dominios de la ingeniería, como por ejemplo la mecánica o la electrónica. Por ello, para interconectar diferentes herramientas de modelado y simulación hace falta el uso de la co-simulación. A pesar de que la co-simulación es una ventaja en términos de flexibilidad para los ingenieros, el uso de diferentes simuladores hace que el tiempo de simulación sea más largo. Por último, al testear sistemas cyberfísicos haciendo uso de simulación, existen diferentes niveles (p.ej., Model, Software y Hardware-in-the-Loop), lo cual incrementa el tiempo para ejecutar casos de test.
Esta tesis tiene como objetivo avanzar en la práctica actual del testeo de sistemas cyber-físicos configurables, proponiendo métodos para la automatización, optimización y depuración. En cuanto a la automatización, primero, se propone una metodología soportada por una herramienta para generar automáticamente instancias de sistemas de test que permiten testear automáticamente configuraciones del sistema CPS configurable (p.ej., haciendo uso de oráculos de test). Segundo, se propone un enfoque para generación de casos de test basado en algoritmos de búsqueda multiobjetivo, los cuales generan un conjunto de casos de test. En cuanto a la optimización, se propone un enfoque para selección y otro para priorización de casos de test, ambos basados en algoritmos de búsqueda, de cara a testear eficientemente sistemas cyberfísicos configurables en diferentes niveles de test. En cuanto a la depuración, se adapta una técnica llamada “Localización de Fallos Basada en Espectro” al contexto de líneas de productos y proponemos un método de aislamiento de fallos. Esto permite localizar bugs no solo en sistemas cyber-físicos configurables sino también en cualquier línea de producto donde se utilicen modelos de características para gestionar la variabilidad.Cyber-Physical Systems (CPSs) integrate digital cyber technologies with physical processes. The variability of these systems is increasing in order to give solution to the different customers demands. As a result, CPSs are becoming configurable or even product lines, which means that they can be set into thousands or millions of configurations. Testing configurable CPSs is a time consuming process, mainly due to the large amount of configurations that need to be tested. The large amount of configurations that need to be tested makes it infeasible to use a prototype of the system. As a result, configurable CPSs are being tested using simulation. However, testing CPSs under simulation is still challenging. First, the simulation time is usually long, since apart of the software, the physical layer needs to be simulated. This physical layer is typically modeled with complex mathematical models, which is computationally very costly. Second, CPSs involve different domains, such as, mechanical and electrical. Engineers of different domains typically employ different tools for modeling their subsystems. As a result, co-simulation is being employed to interconnect different modeling and simulation tools. Despite co-simulation being an advantage in terms of engineers flexibility, the use of different simulation tools makes the simulation time
longer. Lastly, when testing CPSs employing simulation, different test levels exist (i.e., Model, Software and Hardware-in-the-Loop), what increases the time for executing test cases.
This thesis aims at advancing the current practice on testing configurable CPSs by proposing methods for automation, optimization and debugging. Regarding automation, first, we propose a tool supported methodology to automatically generate test system instances that permit automatically testing configurations of the configurable CPS (e.g., by employing test oracles). Second, we propose a test case generation approach based on multi-objective search algorithms that generate cost-effective test suites. As for optimization, we propose a test case selection and a test case prioritization approach, both of them based on search algorithms, to cost-effectively test configurable CPSs at different test levels. Regarding debugging, we adapt a technique
named Spectrum-Based Fault Localization to the product line engineering context and propose a fault isolation method. This permits localizing bugs not only in configurable CPSs but also in any product line where feature models are employed to model variability
Seeding Strategies for Multi-Objective Test Case Selection: An Application on Simulation-based Testing
The time it takes software systems to be tested is usually long. This is often caused by the time it takes the entire test suite to be executed. To optimize this, regression test selection approaches have allowed for improvements to the cost-effectiveness of verification and validation activities in the software industry. In this area, multi-objective algorithms have played a key role in selecting the appropriate subset of test cases from the entire test suite. In this paper, we propose a set of seeding strategies for the test case selection problem that generate the initial population of multi-objective algorithms.We integrated these seeding strategies with an NSGA-II algorithm for solving the test case selection problem in the context of simulation-based testing. We evaluated the strategies with six case studies and a total of 21 fitness combinations for each case study (i.e., a total of 126 problems). Our evaluation suggests that these strategies are indeed helpful for solving the multi-objective test case selection problem. In fact, two of the proposed seeding strategies outperformed the NSGA-II algorithm without seeding population with statistical significance for 92.8 and 96% of the problems
Faster Multi-Goal Simulation-Based Testing Using DoLesS (Domination with Least Square Approximation)
For cyber-physical systems, finding a set of test cases with the least cost
by exploring multiple goals is a complex task. For example, Arrieta et al.
reported that state-of-the-art optimizers struggle to find minimal test suites
for this task. To better manage this task, we propose DoLesS (Domination with
Least Squares Approximation) which uses a domination predicate to sort the
space of possible goals to a small number of representative examples.
Multi-objective domination then divides these examples into a "best" set and
the remaining "rest" set. After that, DoLesS applies an inverted least squares
approximation approach to learn a minimal set of tests that can distinguish
best from rest in the reduced example space. DoLesS has been tested on four
cyber-physical models: a tank flow model; a model of electric car windows; a
safety feature of an AC engine; and a continuous PID controller combined with a
discrete state machine. Comparing to the recent state-of-the-art paper
attempted the same task, DoLesS performs as well or even better as
state-of-the-art, while running 80-360 times faster on average (seconds instead
of hours). Hence, we recommend DoLesSas a fast method to find minimal test
suites for multi-goal cyber-physical systems. For replication purposes, all our
code is on-line:https://github.com/hellonull123/Test_Selection_2021.Comment: 10 pages, 4 figures, 6 tables. Submitted to ICSE 202
Systematic literature review of validation methods for AI systems
Context: Artificial intelligence (AI) has made its way into everyday activities, particularly through new techniques such as machine learning (ML). These techniques are implementable with little domain knowledge. This, combined with the difficulty of testing AI systems with traditional methods, has made system trustworthiness a pressing issue. Objective: This paper studies the methods used to validate practical AI systems reported in the literature. Our goal is to classify and describe the methods that are used in realistic settings to ensure the dependability of AI systems. Method: A systematic literature review resulted in 90 papers. Systems presented in the papers were analysed based on their domain, task, complexity, and applied validation methods. Results: The validation methods were synthesized into a taxonomy consisting of trial, simulation, model-centred validation, and expert opinion. Failure monitors, safety channels, redundancy, voting, and input and output restrictions are methods used to continuously validate the systems after deployment. Conclusions: Our results clarify existing strategies applied to validation. They form a basis for the synthesization, assessment, and refinement of AI system validation in research and guidelines for validating individual systems in practice. While various validation strategies have all been relatively widely applied, only few studies report on continuous validation.Peer reviewe
Acceptance in Incomplete Argumentation Frameworks
A Abstract argumentation frameworks (AFs), originally proposed by Dung, constitute a central formal model for the study of computational aspects of argumentation in AI. Credulous and skeptical acceptance of arguments in a given AF are well-studied problems both in terms of theoretical analysis-especially computational complexity-and the development of practical decision procedures for the problems. However, AFs make the assumption that all attacks between arguments are certain (i.e., present attacks are known to exist, and missing attacks are known to not exist), which can in various settings be a restrictive assumption. A generalization of AFs to incomplete AFs was recently proposed as a formalism that allows the representation of both uncertain attacks and uncertain arguments in AFs. In this article, we explore the impact of allowing for modeling such uncertainties in AFs on the computational complexity of natural generalizations of acceptance problems to incomplete AFs under various central AF semantics. Complementing the complexity-theoretic analysis, we also develop the first practical decision procedures for all of the NP-hard variants of acceptance in incomplete AFs. In terms of complexity analysis, we establish a full complexity landscape, showing that depending on the variant of acceptance and property/semantics, the complexity of acceptance in incomplete AFs ranges from polynomial-time decidable to completeness for Sigma(p)(3). In terms of algorithms, we show through an extensive empirical evaluation that an implementation of the proposed decision procedures, based on boolean satisfiability (SAT) solving, is effective in deciding variants of acceptance under uncertainties. We also establish conditions for what type of atomic changes are guaranteed to be redundant from the perspective of preserving extensions of completions of incomplete AFs, and show that the results allow for considerably improving the empirical efficiency of the proposed SAT-based counterexample-guided abstraction refinement algorithms for acceptance in incomplete AFs for problem variants with complexity beyond NP. (C) 2021 The Authors. Published by Elsevier B.V.Peer reviewe
Impact Assessment of Hypothesized Cyberattacks on Interconnected Bulk Power Systems
The first-ever Ukraine cyberattack on power grid has proven its devastation
by hacking into their critical cyber assets. With administrative privileges
accessing substation networks/local control centers, one intelligent way of
coordinated cyberattacks is to execute a series of disruptive switching
executions on multiple substations using compromised supervisory control and
data acquisition (SCADA) systems. These actions can cause significant impacts
to an interconnected power grid. Unlike the previous power blackouts, such
high-impact initiating events can aggravate operating conditions, initiating
instability that may lead to system-wide cascading failure. A systemic
evaluation of "nightmare" scenarios is highly desirable for asset owners to
manage and prioritize the maintenance and investment in protecting their
cyberinfrastructure. This survey paper is a conceptual expansion of real-time
monitoring, anomaly detection, impact analyses, and mitigation (RAIM) framework
that emphasizes on the resulting impacts, both on steady-state and dynamic
aspects of power system stability. Hypothetically, we associate the
combinatorial analyses of steady state on substations/components outages and
dynamics of the sequential switching orders as part of the permutation. The
expanded framework includes (1) critical/noncritical combination verification,
(2) cascade confirmation, and (3) combination re-evaluation. This paper ends
with a discussion of the open issues for metrics and future design pertaining
the impact quantification of cyber-related contingencies
Alternatives for testing of context-aware software systems in non-academic settings:results from a <i>Rapid Review</i>
Context: Context-awareness challenges the engineering of contemporary software systems and jeopardizes their
testing. The variation of context represents a relevant behavior that deepens the limitations of available software
testing practices and technologies. However, such software systems are mainstream. Therefore, researchers in
non-academic settings also face challenges when developing and testing contemporary software systems.
Objective: To understand how researchers deal with the variation of context when testing context-aware software
systems developed in non-academic settings.
Method: To undertake a secondary study (Rapid Review) to uncover the necessary evidence from primary sources
describing the testing of context-aware software systems outside academia.
Results: The current testing initiatives in non-academic settings aim to generate or improve test suites that can
deal with the context variation and the sheer volume of test input possibilities. They mostly rely on modeling the
systems’ dynamic behavior and increasing computing resources to generate test inputs to achieve this. We found
no evidence of test results aiming at managing context variation through the testing lifecycle process.
Conclusions: So far, the identified testing initiatives and strategies are not ready for mainstream adoption. They
are all domain-specific, and while the ideas and approaches can be reproduced in distinct settings, the technologies are to be re-engineered and tailored to the context-awareness of contemporary software systems in
different problem domains. Further and joint investigations in academia and experiences in non-academic set-
tings can evolve the body of knowledge regarding the testing of contemporary software systems in the field
Machine Learning-based Test Selection for Simulation-based Testing of Self-driving Cars Software
Simulation platforms facilitate the development of emerging Cyber-Physical
Systems (CPS) like self-driving cars (SDC) because they are more efficient and
less dangerous than field operational test cases. Despite this, thoroughly
testing SDCs in simulated environments remains challenging because SDCs must be
tested in a sheer amount of long-running test cases. Past results on software
testing optimization have shown that not all the test cases contribute equally
to establishing confidence in test subjects' quality and reliability, and the
execution of "safe and uninformative" test cases can be skipped to reduce
testing effort. However, this problem is only partially addressed in the
context of SDC simulation platforms. In this paper, we investigate test
selection strategies to increase the cost-effectiveness of simulation-based
testing in the context of SDCs. We propose an approach called SDC-Scissor (SDC
coSt-effeCtIve teSt SelectOR) that leverages Machine Learning (ML) strategies
to identify and skip test cases that are unlikely to detect faults in SDCs
before executing them.
Our evaluation shows that SDC-Scissor outperforms the baselines. With the
Logistic model, we achieve an accuracy of 70%, a precision of 65%, and a recall
of 80% in selecting tests leading to a fault and improved testing
cost-effectiveness. Specifically, SDC-Scissor avoided the execution of 50% of
unnecessary tests as well as outperformed two baseline strategies.
Complementary to existing work, we also integrated SDC-Scissor into the context
of an industrial organization in the automotive domain to demonstrate how it
can be used in industrial settings.Comment: arXiv admin note: substantial text overlap with arXiv:2111.0466
- …