647 research outputs found
Using Machine Learning to Generate Test Oracles: A Systematic Literature Review
Machine learning may enable the automated generation of test oracles. We have
characterized emerging research in this area through a systematic literature
review examining oracle types, researcher goals, the ML techniques applied, how
the generation process was assessed, and the open research challenges in this
emerging field.
Based on a sample of 22 relevant studies, we observed that ML algorithms
generated test verdict, metamorphic relation, and - most commonly - expected
output oracles. Almost all studies employ a supervised or semi-supervised
approach, trained on labeled system executions or code metadata - including
neural networks, support vector machines, adaptive boosting, and decision
trees. Oracles are evaluated using the mutation score, correct classifications,
accuracy, and ROC. Work-to-date show great promise, but there are significant
open challenges regarding the requirements imposed on training data, the
complexity of modeled functions, the ML algorithms employed - and how they are
applied - the benchmarks used by researchers, and replicability of the studies.
We hope that our findings will serve as a roadmap and inspiration for
researchers in this field.Comment: Pre-print. Article accepted to 1st International Workshop on Test
Oracles at ESEC/FSE 202
Using Machine Learning to Generate Test Oracles: A Systematic Literature Review
Machine learning may enable the automated generation of test oracles. We have characterized emerging research in this area through a systematic literature review examining oracle types, researcher goals, the ML techniques applied, how the generation process was assessed, and the open research challenges in this emerging field.Based on a sample of 22 relevant studies, we observed that ML algorithms generated test verdict, metamorphic relation, and - most commonly - expected output oracles. Almost all studies employ a supervised or semi-supervised approach, trained on labeled system executions or code metadata - including neural networks, support vector machines, adaptive boosting, and decision trees. Oracles are evaluated using the mutation score, correct classifications, accuracy, and ROC. Work-to-date show great promise, but there are significant open challenges regarding the requirements imposed on training data, the complexity of modeled functions, the ML algorithms employed - and how they are applied - the benchmarks used by researchers, and replicability of the studies. We hope that our findings will serve as a roadmap and inspiration for researchers in this field
The Oracle Problem in Software Testing: A Survey
Testing involves examining the behaviour of a system in order to discover potential faults. Given an input for a system, the challenge of distinguishing the corresponding desired, correct behaviour from potentially incorrect behavior is called the “test oracle problem”. Test oracle automation is important to remove a current bottleneck that inhibits greater overall test automation. Without test oracle automation, the human has to determine whether observed behaviour is correct. The literature on test oracles has introduced techniques for oracle automation, including modelling, specifications, contract-driven development and metamorphic testing. When none of these is completely adequate, the final source of test oracle information remains the human, who may be aware of informal specifications, expectations, norms and domain specific information that provide informal oracle guidance. All forms of test oracles, even the humble human, involve challenges of reducing cost and increasing benefit. This paper provides a comprehensive survey of current approaches to the test oracle problem and an analysis of trends in this important area of software testing research and practice
The Integration of Machine Learning into Automated Test Generation: A Systematic Mapping Study
Context: Machine learning (ML) may enable effective automated test
generation.
Objective: We characterize emerging research, examining testing practices,
researcher goals, ML techniques applied, evaluation, and challenges.
Methods: We perform a systematic mapping on a sample of 102 publications.
Results: ML generates input for system, GUI, unit, performance, and
combinatorial testing or improves the performance of existing generation
methods. ML is also used to generate test verdicts, property-based, and
expected output oracles. Supervised learning - often based on neural networks -
and reinforcement learning - often based on Q-learning - are common, and some
publications also employ unsupervised or semi-supervised learning.
(Semi-/Un-)Supervised approaches are evaluated using both traditional testing
metrics and ML-related metrics (e.g., accuracy), while reinforcement learning
is often evaluated using testing metrics tied to the reward function.
Conclusion: Work-to-date shows great promise, but there are open challenges
regarding training data, retraining, scalability, evaluation complexity, ML
algorithms employed - and how they are applied - benchmarks, and replicability.
Our findings can serve as a roadmap and inspiration for researchers in this
field.Comment: Under submission to Software Testing, Verification, and Reliability
journal. (arXiv admin note: text overlap with arXiv:2107.00906 - This is an
earlier study that this study extends
Automated Misconfiguration Repair of Configurable Cyber-Physical Systems with Search: an Industrial Case Study on Elevator Dispatching Algorithms
Real-world Cyber-Physical Systems (CPSs) are usually configurable. Through
parameters, it is possible to configure, select or unselect different system
functionalities. While this provides high flexibility, it also becomes a source
for failures due to misconfigurations. The large number of parameters these
systems have and the long test execution time in this context due to the use of
simulation-based testing make the manual repair process a cumbersome activity.
Subsequently, in this context, automated repairing methods are paramount. In
this paper, we propose an approach to automatically repair CPSs'
misconfigurations. Our approach is evaluated with an industrial CPS case study
from the elevation domain. Experiments with a real building and data obtained
from operation suggests that our approach outperforms a baseline algorithm as
well as the state of the practice (i.e., manual repair carried out by domain
experts).Comment: To be published in the 45th International Conference on Software
Engineering, SEIP trac
Automatically generating complex test cases from simple ones
While source code expresses and implements design considerations for software system, test cases capture and represent the domain knowledge of software developer, her assumptions on the implicit and explicit interaction protocols in the system, and the expected behavior of different modules of the system in normal and exceptional conditions. Moreover, test cases capture information about the environment and the data the system operates on. As such, together with the system source code, test cases integrate important system and domain knowledge. Besides being an important project artifact, test cases embody up to the half the overall software development cost and effort. Software projects produce many test cases of different kind and granularity to thoroughly check the system functionality, aiming to prevent, detect, and remove different types of faults. Simple test cases exercise small parts of the system aiming to detect faults in single modules. More complex integration and system test cases exercise larger parts of the system aiming to detect problems in module interactions and verify the functionality of the system as a whole. Not surprisingly, the test case complexity comes at a cost -- developing complex test cases is a laborious and expensive task that is hard to automate. Our intuition is that important information that is naturally present in test cases can be reused to reduce the effort in generation of new test cases. This thesis develops this intuition and investigates the phenomenon of information reuse among test cases. We first empirically investigated many test cases from real software projects and demonstrated that test cases of different granularity indeed share code fragments and build upon each other. Then we proposed an approach for automatically generating complex test cases by extracting and exploiting information in existing simple ones. In particular, our approach automatically generates integration test cases from unit ones. We implemented our approach in a prototype to evaluate its ability to generate new and useful test cases for real software systems. Our studies show that test cases generated with our approach reveal new interaction faults even in well tested applications. We evaluated the effectiveness of our approach by comparing it with the state of the art test generation techniques. The evaluation results show that our approach is effective, it finds relevant faults differently from other approaches that tend to find different and usually less relevant faults
RMT: Rule-based Metamorphic Testing for Autonomous Driving Models
Deep neural network models are widely used for perception and control in
autonomous driving. Recent work uses metamorphic testing but is limited to
using equality-based metamorphic relations and does not provide expressiveness
for defining inequality-based metamorphic relations. To encode real world
traffic rules, domain experts must be able to express higher order relations
e.g., a vehicle should decrease speed in certain ratio, when there is a vehicle
x meters ahead and compositionality e.g., a vehicle must have a larger
deceleration, when there is a vehicle ahead and when the weather is rainy and
proportional compounding effect to the test outcome. We design RMT, a
declarative rule-based metamorphic testing framework. It provides three
components that work in concert:(1) a domain specific language that enables an
expert to express higher-order, compositional metamorphic relations, (2)
pluggable transformation engines built on a variety of image and graphics
processing techniques, and (3) automated test generation that translates a
human-written rule to a corresponding executable, metamorphic relation and
synthesizes meaningful inputs.Our evaluation using three driving models shows
that RMT can generate meaningful test cases on which 89% of erroneous
predictions are found by enabling higher-order metamorphic relations.
Compositionality provides further aids for generating meaningful, synthesized
inputs-3012 new images are generated by compositional rules. These detected
erroneous predictions are manually examined and confirmed by six human judges
as meaningful traffic rule violations. RMT is the first to expand automated
testing capability for autonomous vehicles by enabling easy mapping of traffic
regulations to executable metamorphic relations and to demonstrate the benefits
of expressivity, customization, and pluggability
Performance-Driven Metamorphic Testing of Cyber-Physical Systems
Cyber-physical systems (CPSs) are a new generation
of systems, which integrate software with physical processes. The
increasing complexity of these systems, combined with the un certainty in their interactions with the physical world, makes the
definition of effective test oracles especially challenging, facing the
well-known test oracle problem. Metamorphic testing has shown
great potential to alleviate the test oracle problem by exploiting the
relations among the inputs and outputs of different executions of
the system, so-called metamorphic relations (MRs). In this article,
we propose an MR pattern called PV for the identification of
performance-driven MRs, and we show its applicability in two
CPSs from different domains, which are automated navigation
systems and elevator control systems. For the evaluation, we as sessed the effectiveness of this approach for detecting failures in an
open-source simulation-based autonomous navigation system, as
well as in an industrial case study from the elevation domain. We
derive concrete MRs based on the PV pattern for both case studies,
and we evaluate their effectiveness with seeded faults. Results show
that the approach is effective at detecting over 88% of the seeded
faults, while keeping the ratio of FPs at 4% or lower.European Union's Horizon 2020 Research and Innovation Programme (Grant Number: 871319)Junta de Andalucía US-1264651 (APOLO)Junta de Andalucía P18-FR-2895 (EKIPMENT-PLUS)Ministerio de Ciencia e Innovación RTI2018-101204-B-C21 (HORATIO)Mondragon Unibertsitatea IT1519-2
RobotChain: Artificial Intelligence on a Blockchain using Tezos Technology
Blockchain technology is not only growing everyday at a fast-passed rhythm, but it is also a disruptive technology that has changed how we look at financial transactions. By providing a way
to trust an unknown network and by allowing us to conduct transactions without the need for a
central authority, blockchain has grown exponentially. Moreover, blockchain also provides decentralization of the data, immutability, accessibility, non-repudiation and irreversibility properties that makes this technology a must in many industries. But, even thought blockchain
provides interesting properties, it has not been extensively used outside the financial scope.
Similarly, robots have been increasingly used in factories to automate tasks that range from
picking objects, to transporting them and also to work collaboratively with humans to perform
complex tasks. It is important to enforce that robots act between legal and moral boundaries
and that their events and data are securely stored and auditable. This rarely happens, as robots
are programmed to do a specific task without certainty that that task will always be performed
correctly and their data is either locally stored, without security measures, or disregarded. This
means that the data, especially logs, can be altered, which means that robots and manufacturers can be accused of problems that they did not cause. Henceforth, in this work, we sought
to integrate blockchain with robotics with the goal to provide enhanced security to robots, to
the data and to leverage artificial intelligence algorithms. By doing an extensive overview of
the methods that integrate blockchain and artificial intelligence or robotics, we found that this
is a growing field but there is a lack of proposals that try to improve robotic systems by using
blockchain. It was also clear that most of the existing proposals that integrate artificial intelligence and blockchain, are focused on building marketplaces and only use the latter to storage
transactions. So, in this document, we proposed three different methods that use blockchain
to solve different problems associated with robots. The first one is a method to securely store
robot logs in a blockchain by using smart-contracts as storage and automatically detect when
anomalies occur in a robot by using the data contained in the blockchain and a smart-contract.
By using smart-contracts, it is assured that the data is secure and immutable as long as the
blockchain has enough peers to participate in the consensus process. The second method goes
beyond registering events to also register information about external sensors, like a camera,
and by using smart-contracts to allow Oracles to interact with the blockchain, it was possible to
leverage image analysis algorithms that can detect the presence of material to be picked. This
information is then inserted into a smart-contract that automatically defines the movement that
a robot should have, regarding the number of materials present to be picked. The third proposal
is a method that uses blockchain to store information about the robots and the images derived
from a Kinect. This information is then used by Oracles that check if there is any person located
inside a robot workspace. If there is any, this information is stored and different Oracles try to
identify the person. Then, a smart-contract acts appropriately by changing or even stopping the
robot depending on the identity of the person and if the person is located inside the warning or
the critical zone surrounding the robot.
With this work, we show how blockchain can be used in robotic environments and how it
can beneficial in contexts where multi-party cooperation, security, and decentralization of the
data is essential. We also show how Oracles can interact with the blockchain and distributively
cooperate to leverage artificial intelligence algorithms to perform analysis in the data that
allow us to detect robotic anomalies, material in images and the presence of people. We also show that smart-contracts can be used to perform more tasks than just serve the purpose of
automatically do monetary transactions. The proposed architectures are modular and can be
used in multiple contexts such as in manufacturing, network control, robot control, and others
since they are easy to integrate, adapt, maintain and extend to new domains. We expect
that the intersection of blockchain and robotics will shape part of the future of robotics once
blockchain is more widely used and easy to integrate. This integration will be very prominent
in tasks where robots need to behave under certain constraints, in swarm robotics due to the
fact that blockchain offers global information and in factories because the actions undertaken
by a robot can easily be extended to the rest of the robots by using smart-contracts.Hoje em dia é possível ver que a blockchain não está apenas a crescer a um ritmo exponencial, mas que é também uma tecnologia disruptiva que mudou a forma como trabalhamos com
transações financeiras. Ao fornecer uma maneira eficiente de confiar numa rede desconhecida
e de permitir realizar transações sem a necessidade de uma autoridade central, a blockchain
cresceu rapidamente. Além disso, a blockchain fornece também descentralização de dados,
imutabilidade, acessibilidade, não-repúdio e irreversibilidade, o que torna esta tecnologia indispensável em muitos setores. Mas, mesmo fornecendo propriedades interessantes, a blockchain não tem sido amplamente utilizada fora do âmbito financeiro. Da mesma forma, os robôs
têm sido cada vez mais utilizados em fábricas para automatizar tarefas que vão desde pegar
objetos, transportá-los e colaborar com humanos para realizar tarefas complexas. Porém, é
importante impor que os robôs atuem entre certos limites legais e morais e que seus eventos
e dados são armazenados com segurança e que estes possam ser auditáveis. O problema é que
isso raramente acontece. Os robôs são programados para executar uma tarefa específica sem
se ter total certeza de que essa tarefa irá ser executada sempre de maneira correta, e os seus
dados são armazenados localmente, desconsiderando a segurança dos dados. Sendo que em
muitas ocasiões, não existe qualquer segurança. Isso significa que os dados, especialmente os
logs, podem ser alterados, o que pode resultar em que os robôs e, pela mesma linha de pensamento, os fabricantes, possam ser acusados de problemas que não causaram. Tendo isto em
consideração, neste trabalho, procuramos integrar a blockchain com a robótica, com o objetivo
de proporcionar maior segurança aos robôs e aos dados que geram e potenciar ainda a utilização de algoritmos de inteligência artificial. Fazendo uma visão abrangente dos métodos que
propõem integrar a blockchain e inteligência artificial ou robótica, descobrimos que este é um
campo em crescimento, mas que há uma falta de propostas que tentem melhorar os sistemas
robóticos utilizando a blockchain. Ficou também claro que a maioria das propostas existentes
que integram inteligência artificial e blockchain estão focadas na construção de marketplaces e
só utilizam a blockchain para armazenar a informação sobre as transações que foram executadas. Assim, neste documento, propomos três métodos que utilizam a blockchain para resolver
diferentes problemas associados a robôs. O primeiro é um método para armazenar, com segurança, logs de robôs dentro de uma blockchain, utilizando para isso smart-contracts como
armazenamento. Neste método foi também proposta uma maneira de detetar anomalias em
robôs automaticamente, utilizando para isso os dados contidos na blockchain e smart-contracts
para definir a lógica do algoritmo. Ao utilizar smart-contracts, é garantido que os dados são seguros e imutáveis, desde que a blockchain contenha nós suficientes a participar no algoritmo de
consenso. O segundo método vai além de registar eventos, para registar também informações
sobre sensores externos, como uma câmara, e utilizando smart-contracts para permitir que Óraculos interajam com a blockchain, foi possível utilizar algoritmos de análise de imagens, que
podem detetar a presença de material para ser recolhido. Esta informação é então inserida
num smart-contract que define automaticamente o movimento que um robô deve ter, tendo
em consideração a quantidade de material à espera para ser recolhida. A terceira proposta é
um método que utiliza a blockchain para armazenar informações sobre robôs, e imagens provenientes de uma Kinect. Esta informação é então utilizada por Óraculos que verificam se existe
alguma pessoa dentro do um espaço de trabalho de um robô. Se existir alguém, essa informação
é armazenada e diferentes Óraculos tentam identificar a pessoa. No fim, um smart-contract
age apropriadamente, mudando ou até mesmo parando o robô, dependendo da identidade da Com este trabalho, mostramos como a blockchain pode ser utilizada em ambientes onde existam robôs e como esta pode ser benéfica em contextos onde a cooperação entre várias entidades, a segurança e a descentralização dos dados são essenciais. Mostramos também como
Óraculos podem interagir com a blockchain e cooperar de forma distribuída, para alavancar
algoritmos de inteligência artificial de forma a realizar análises nos dados, o que nos permite
detetar anomalias robóticas, material para ser recolhido e a presença de pessoas em imagens.
Mostramos também que os smart-contracts podem ser utilizados para executar mais tarefas do
que servir o propósito de fazer transações monetárias de forma automática. As arquiteturas
propostas neste trabalho são modulares e podem ser utilizadas em vários contextos, como no
fabrico de peças, controle de robô e outras. Devido ao facto de que as arquiteturas propostas,
são fáceis de integrar, adaptar, manter e estender a novos domínios. A nossa opinião é que a
interseção entre a blockchain e a robótica irá moldar parte do futuro da robótica moderna assim
que a blockchain seja mais utilizada e fácil de integrar em sistemas robóticos. Esta integração
será muito proeminente em tarefas onde os robôs precisam de se comportar sob certas restrições, em enxames de robôs, devido ao fato de que a blockchain fornece informação global sobre
o estado da rede, e também em fábricas, porque as ações realizadas por um robô podem ser
facilmente estendidas ao resto dos robôs, e porque fornece um mecanismo extra de segurança
aos dados e a todas as ações que são efetuadas com ajuda de smart-contracts
- …