82 research outputs found
Deep specification mining
Singapore National Research Foundatio
Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing
We have recently witnessed tremendous success of Machine Learning (ML) in
practical applications. Computer vision, speech recognition and language
translation have all seen a near human level performance. We expect, in the
near future, most business applications will have some form of ML. However,
testing such applications is extremely challenging and would be very expensive
if we follow today's methodologies. In this work, we present an articulation of
the challenges in testing ML based applications. We then present our solution
approach, based on the concept of Metamorphic Testing, which aims to identify
implementation bugs in ML based image classifiers. We have developed
metamorphic relations for an application based on Support Vector Machine and a
Deep Learning based application. Empirical validation showed that our approach
was able to catch 71% of the implementation bugs in the ML applications.Comment: Published at 27th ACM SIGSOFT International Symposium on Software
Testing and Analysis (ISSTA 2018
ItyFuzz: Snapshot-Based Fuzzer for Smart Contract
Smart contracts are critical financial instruments, and their security is of
utmost importance. However, smart contract programs are difficult to fuzz due
to the persistent blockchain state behind all transactions. Mutating sequences
of transactions are complex and often lead to a suboptimal exploration for both
input and program spaces. In this paper, we introduce a novel snapshot-based
fuzzer ItyFuzz for testing smart contracts. In ItyFuzz, instead of storing
sequences of transactions and mutating from them, we snapshot states and
singleton transactions. To explore interesting states, ItyFuzz introduces a
dataflow waypoint mechanism to identify states with more potential momentum.
ItyFuzz also incorporates comparison waypoints to prune the space of states. By
maintaining snapshots of the states, ItyFuzz can synthesize concrete exploits
like reentrancy attacks quickly. Because ItyFuzz has second-level response time
to test a smart contract, it can be used for on-chain testing, which has many
benefits compared to local development testing. Finally, we evaluate ItyFuzz on
real-world smart contracts and some hacked on-chain DeFi projects. ItyFuzz
outperforms existing fuzzers in terms of instructional coverage and can find
and generate realistic exploits for on-chain projects quickly.Comment: ISSTA 202
Test Case Prioritization for Acceptance Testing of Cyber Physical Systems: A Multi-Objective Search-Based Approach
Acceptance testing validates that a system meets its requirements and determines whether it can be sufficiently trusted and put into operation. For cyber physical systems (CPS), acceptance testing is a hardware-in-the-loop process conducted in a (near-)operational environment. Acceptance testing of a CPS often necessitates that the test cases be prioritized, as there are usually too many scenarios to consider given time constraints. CPS acceptance testing is further complicated by the uncertainty in the environment and the impact of testing on hardware. We propose an automated test case prioritization approach for CPS acceptance testing, accounting for time budget constraints, uncertainty, and hardware damage risks. Our approach is based on multi-objective search, combined with a test case minimization algorithm that eliminates redundant operations from an ordered sequence of test cases. We evaluate our approach on a representative case study from the satellite domain. The results indicate that, compared to test cases that are prioritized manually by satellite engineers, our automated approach more than doubles the number of test cases that fit into a given time frame, while reducing to less than one third the number of operations that entail the risk of damage to key hardware components
Are Formal Contracts a useful Digital Twin of Software Systems?
Digital Twins are a trend topic in the industry today to either manage runtime information or forecast properties of devices and products. The techniques for Digitial Twins are already employed in several disciplines of formal methods, in particular, formal verification, runtime verification and specification inference. In this paper, we connect the Digital Twin concept and existing research areas in the field of formal methods. We sketch how digital twins for software-centric systems can be forged from existing formal methods
- …