1,524 research outputs found
JUGE: An Infrastructure for Benchmarking Java Unit Test Generators
Researchers and practitioners have designed and implemented various automated
test case generators to support effective software testing. Such generators
exist for various languages (e.g., Java, C#, or Python) and for various
platforms (e.g., desktop, web, or mobile applications). Such generators exhibit
varying effectiveness and efficiency, depending on the testing goals they aim
to satisfy (e.g., unit-testing of libraries vs. system-testing of entire
applications) and the underlying techniques they implement. In this context,
practitioners need to be able to compare different generators to identify the
most suited one for their requirements, while researchers seek to identify
future research directions. This can be achieved through the systematic
execution of large-scale evaluations of different generators. However, the
execution of such empirical evaluations is not trivial and requires a
substantial effort to collect benchmarks, setup the evaluation infrastructure,
and collect and analyse the results. In this paper, we present our JUnit
Generation benchmarking infrastructure (JUGE) supporting generators (e.g.,
search-based, random-based, symbolic execution, etc.) seeking to automate the
production of unit tests for various purposes (e.g., validation, regression
testing, fault localization, etc.). The primary goal is to reduce the overall
effort, ease the comparison of several generators, and enhance the knowledge
transfer between academia and industry by standardizing the evaluation and
comparison process. Since 2013, eight editions of a unit testing tool
competition, co-located with the Search-Based Software Testing Workshop, have
taken place and used and updated JUGE. As a result, an increasing amount of
tools (over ten) from both academia and industry have been evaluated on JUGE,
matured over the years, and allowed the identification of future research
directions
Modeling and Testing Implementations of Protocols with Complex Messages
This paper presents a new language called APSL for formally describing
protocols to facilitate automated testing. Many real world communication
protocols exchange messages whose structures are not trivial, e.g. they may
consist of multiple and nested fields, some could be optional, and some may
have values that depend on other fields. To properly test implementations of
such a protocol, it is not sufficient to only explore different orders of
sending and receiving messages. We also need to investigate if the
implementation indeed produces correctly formatted messages, and if it responds
correctly when it receives different variations of every message type. APSL's
main contribution is its sublanguage that is expressive enough to describe
complex message formats, both text-based and binary. As an example, this paper
also presents a case study where APSL is used to model and test a subset of
Courier IMAP email server
Combining Static and Dynamic Analysis for Vulnerability Detection
In this paper, we present a hybrid approach for buffer overflow detection in
C code. The approach makes use of static and dynamic analysis of the
application under investigation. The static part consists in calculating taint
dependency sequences (TDS) between user controlled inputs and vulnerable
statements. This process is akin to program slice of interest to calculate
tainted data- and control-flow path which exhibits the dependence between
tainted program inputs and vulnerable statements in the code. The dynamic part
consists of executing the program along TDSs to trigger the vulnerability by
generating suitable inputs. We use genetic algorithm to generate inputs. We
propose a fitness function that approximates the program behavior (control
flow) based on the frequencies of the statements along TDSs. This runtime
aspect makes the approach faster and accurate. We provide experimental results
on the Verisec benchmark to validate our approach.Comment: There are 15 pages with 1 figur
DeepSQLi: Deep Semantic Learning for Testing SQL Injection
Security is unarguably the most serious concern for Web applications, to
which SQL injection (SQLi) attack is one of the most devastating attacks.
Automatically testing SQLi vulnerabilities is of ultimate importance, yet is
unfortunately far from trivial to implement. This is because the existence of a
huge, or potentially infinite, number of variants and semantic possibilities of
SQL leading to SQLi attacks on various Web applications. In this paper, we
propose a deep natural language processing based tool, dubbed DeepSQLi, to
generate test cases for detecting SQLi vulnerabilities. Through adopting deep
learning based neural language model and sequence of words prediction, DeepSQLi
is equipped with the ability to learn the semantic knowledge embedded in SQLi
attacks, allowing it to translate user inputs (or a test case) into a new test
case, which is semantically related and potentially more sophisticated.
Experiments are conducted to compare DeepSQLi with SQLmap, a state-of-the-art
SQLi testing automation tool, on six real-world Web applications that are of
different scales, characteristics and domains. Empirical results demonstrate
the effectiveness and the remarkable superiority of DeepSQLi over SQLmap, such
that more SQLi vulnerabilities can be identified by using a less number of test
cases, whilst running much faster
A Methodological Framework for Evaluating Software Testing Techniques and Tools
© 2012 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.There exists a real need in industry to have guidelines on what testing techniques use for different testing objectives, and how usable (effective, efficient, satisfactory) these techniques are. Up to date, these guidelines do not exist. Such guidelines could be obtained by doing secondary studies on a body of evidence consisting of case studies evaluating and comparing testing techniques and tools. However, such a body of evidence is also lacking. In this paper, we will make a first step towards creating such body of evidence by defining a general methodological evaluation framework that can simplify the design of case studies for comparing software testing tools, and make the results more precise, reliable, and easy to compare. Using this framework, (1) software testing practitioners can more easily define case studies through an instantiation of the framework, (2) results can be better compared since they are all executed according to a similar design, (3) the gap in existing work on methodological evaluation frameworks will be narrowed, and (4) a body of evidence will be initiated. By means of validating the framework, we will present successful applications of this methodological framework to various case studies for evaluating testing tools in an industrial environment with real objects and real subjects.This work was funded by the European project FITTEST
(ICT257574, 2010-2013) and Spanish National project
CaSA-Calidad (TIN2010-12312-E, Ministerio de Ciencia e
Innovación)Vos, TE.; MarÃn, B.; Escalona, MJ.; Marchetto, A. (2012). A Methodological Framework for Evaluating Software Testing Techniques and Tools. IEEE. https://doi.org/10.1109/QSIC.2012.16
Using Executable VDM++ Models in an Industrial Application - Self-defense System for Fighter Aircraft
When developing complex software systems, one of the most significant challenges is to make sure that the customer and developer agree on the requirements of the system. By using executable models early in the development process, a higher degree of confidence can be gained in the system design and misunderstandings or ambiguous functional requirements can be avoided. This paper presents an industrial case of a communication protocol between two parts of a selfdefense system used on-board fighter aircraft. An executable model of both systems were created using the Vienna Development Method (VDM), and exercised using many scenarios to cover different corner cases. This was done as an alternative to analysing all the scenarios by hand, which would be much more time consuming and far more error prone. The results of the scenario based tests were used to communicate with the customer and ensure that agreement of the requirements was reached
- …