6,618 research outputs found
The Integration of Machine Learning into Automated Test Generation: A Systematic Mapping Study
Context: Machine learning (ML) may enable effective automated test
generation.
Objective: We characterize emerging research, examining testing practices,
researcher goals, ML techniques applied, evaluation, and challenges.
Methods: We perform a systematic mapping on a sample of 102 publications.
Results: ML generates input for system, GUI, unit, performance, and
combinatorial testing or improves the performance of existing generation
methods. ML is also used to generate test verdicts, property-based, and
expected output oracles. Supervised learning - often based on neural networks -
and reinforcement learning - often based on Q-learning - are common, and some
publications also employ unsupervised or semi-supervised learning.
(Semi-/Un-)Supervised approaches are evaluated using both traditional testing
metrics and ML-related metrics (e.g., accuracy), while reinforcement learning
is often evaluated using testing metrics tied to the reward function.
Conclusion: Work-to-date shows great promise, but there are open challenges
regarding training data, retraining, scalability, evaluation complexity, ML
algorithms employed - and how they are applied - benchmarks, and replicability.
Our findings can serve as a roadmap and inspiration for researchers in this
field.Comment: Under submission to Software Testing, Verification, and Reliability
journal. (arXiv admin note: text overlap with arXiv:2107.00906 - This is an
earlier study that this study extends
Recommended from our members
Using formal methods to support testing
Formal methods and testing are two important approaches that assist in the development of high quality software. While traditionally these approaches have been seen as rivals, in recent
years a new consensus has developed in which they are seen as complementary. This article reviews the state of the art regarding ways in which the presence of a formal specification can be used to assist testing
Automated test of evolving software
A thesis submitted to the University of Luton, in partial fulfilment of the requirements for the degree of Doctor of PhilosophyComputers and the software they run are pervasive, yet released software is often unreliable, which has many consequences. Loss of time and earnings can be caused by application software (such as word processors) behaving incorrectly or crashing. Serious disruption can occur as in the l4th August 2003 blackouts in North East USA and Canadal, or serious injury or death can be caused as in the Therac-25 overdose incidents.
One way to improve the quality of software is to test it thoroughly. However, software testing is time consuming, the resources, capabilities and skills needed to carry it out are often not available and the time required is often curtailed because of pressures to meet delivery deadlines3. Automation should allow more thorough testing in the time available and improve the quality of delivered software, but there are some problems with automation that this research addresses.
Firstly, it is difficult to determine ifthe system under test (SUT) has passed or failed a test. This is known as the oracle problem4 and is often ignored in software testing research. Secondly, many software development organisations use an iterative and incremental process, known as evolutionary development, to write software. Following release, software continues evolving as customers demand new features and improvements to existing ones5. This evolution means that automated test suites must be maintained throughout the life ofthe software.
A contribution of this research is a methodology that addresses automatic generation of the test cases, execution of the test cases and evaluation of the outcomes from running each test.
"Predecessor" software is used to solve the oracle problem. This is software that already exists, such as a previous version of evolving software, or software from a different vendor that solves the same, or similar, problems. However, the resulting oracle is assumed not be perfect, so rules are defined in an interface, which are used by the evaluator in the test evaluation stage to handle the expected differences.
The interface also specifies functional inputs and outputs to the SUT. An algorithm has been developed that creates a Markov Chain Transition Matrix (MCTM) model of the SUT from the interface. Tests are then generated automatically by making a random walk of the MCTM. This means that instead of maintaining a large suite of tests, or a large model of the SUT, only the interface needs to be maintained.
1) NERC Steering Group (2004). Technical Analysis ofthe August 14,2003, Blackout: What
Happened, Why, and What Did We Learn? July 13th 2004. Available from:
ftp:/ /www.nerc.com/pub/sys/all_ updl/docslblackoutINERC ]inatBlackout_Report _ 07_13_ 04.pdf
2) Leveson N. G., Turner C. S. (1993) An investigation of the Therac-25 accidents. IEEE Computer,
Vo126, No 7, Pages 18-41.
3) LogicaCMG (2005) Testing Times for Board Rooms. Available from
http://www.logicacmg.com/pdf/trackeditestingTimesBoardRooms.pdf
4) Bertolino, A. (2003) Software Testing Research and Practice, ASM 2003, Lecture Notes in
Computer Science, Vol 2589, Pages 1-21.
5) Sommerville, 1. (2004) Software Engineering, 7th Edition. Addison Wesley. ISBN 0-321-21026-3
Stacco: Differentially Analyzing Side-Channel Traces for Detecting SSL/TLS Vulnerabilities in Secure Enclaves
Intel Software Guard Extension (SGX) offers software applications enclave to
protect their confidentiality and integrity from malicious operating systems.
The SSL/TLS protocol, which is the de facto standard for protecting
transport-layer network communications, has been broadly deployed for a secure
communication channel. However, in this paper, we show that the marriage
between SGX and SSL may not be smooth sailing.
Particularly, we consider a category of side-channel attacks against SSL/TLS
implementations in secure enclaves, which we call the control-flow inference
attacks. In these attacks, the malicious operating system kernel may perform a
powerful man-in-the-kernel attack to collect execution traces of the enclave
programs at page, cacheline, or branch level, while positioning itself in the
middle of the two communicating parties. At the center of our work is a
differential analysis framework, dubbed Stacco, to dynamically analyze the
SSL/TLS implementations and detect vulnerabilities that can be exploited as
decryption oracles. Surprisingly, we found exploitable vulnerabilities in the
latest versions of all the SSL/TLS libraries we have examined.
To validate the detected vulnerabilities, we developed a man-in-the-kernel
adversary to demonstrate Bleichenbacher attacks against the latest OpenSSL
library running in the SGX enclave (with the help of Graphene) and completely
broke the PreMasterSecret encrypted by a 4096-bit RSA public key with only
57286 queries. We also conducted CBC padding oracle attacks against the latest
GnuTLS running in Graphene-SGX and an open-source SGX-implementation of mbedTLS
(i.e., mbedTLS-SGX) that runs directly inside the enclave, and showed that it
only needs 48388 and 25717 queries, respectively, to break one block of AES
ciphertext. Empirical evaluation suggests these man-in-the-kernel attacks can
be completed within 1 or 2 hours.Comment: CCS 17, October 30-November 3, 2017, Dallas, TX, US
Cloud engineering is search based software engineering too
Many of the problems posed by the migration of computation to cloud platforms can be formulated and solved using techniques associated with Search Based Software Engineering (SBSE). Much of cloud software engineering involves problems of optimisation: performance, allocation, assignment and the dynamic balancing of resources to achieve pragmatic trade-offs between many competing technical and business objectives. SBSE is concerned with the application of computational search and optimisation to solve precisely these kinds of software engineering challenges. Interest in both cloud computing and SBSE has grown rapidly in the past five years, yet there has been little work on SBSE as a means of addressing cloud computing challenges. Like many computationally demanding activities, SBSE has the potential to benefit from the cloud; ‘SBSE in the cloud’. However, this paper focuses, instead, of the ways in which SBSE can benefit cloud computing. It thus develops the theme of ‘SBSE for the cloud’, formulating cloud computing challenges in ways that can be addressed using SBSE
- …