414 research outputs found
Automatic Software Repair: a Bibliography
This article presents a survey on automatic software repair. Automatic
software repair consists of automatically finding a solution to software bugs
without human intervention. This article considers all kinds of repairs. First,
it discusses behavioral repair where test suites, contracts, models, and
crashing inputs are taken as oracle. Second, it discusses state repair, also
known as runtime repair or runtime recovery, with techniques such as checkpoint
and restart, reconfiguration, and invariant restoration. The uniqueness of this
article is that it spans the research communities that contribute to this body
of knowledge: software engineering, dependability, operating systems,
programming languages, and security. It provides a novel and structured
overview of the diversity of bug oracles and repair operators used in the
literature
Metamorphic testing: a review of challenges and opportunities
Metamorphic testing is an approach to both test case generation and test result verification. A central element is a set of metamorphic relations, which are necessary properties of the target function or algorithm in relation to multiple inputs and their expected outputs. Since its first publication, we have witnessed a rapidly increasing body of work examining metamorphic testing from various perspectives, including metamorphic relation identification, test case generation, integration with other software engineering techniques, and the validation and evaluation of software systems. In this paper, we review the current research of metamorphic testing and discuss the challenges yet to be addressed. We also present visions for further improvement of metamorphic testing and highlight opportunities for new research
Automated Testing and Improvement of Named Entity Recognition Systems
Named entity recognition (NER) systems have seen rapid progress in recent
years due to the development of deep neural networks. These systems are widely
used in various natural language processing applications, such as information
extraction, question answering, and sentiment analysis. However, the complexity
and intractability of deep neural networks can make NER systems unreliable in
certain circumstances, resulting in incorrect predictions. For example, NER
systems may misidentify female names as chemicals or fail to recognize the
names of minority groups, leading to user dissatisfaction. To tackle this
problem, we introduce TIN, a novel, widely applicable approach for
automatically testing and repairing various NER systems. The key idea for
automated testing is that the NER predictions of the same named entities under
similar contexts should be identical. The core idea for automated repairing is
that similar named entities should have the same NER prediction under the same
context. We use TIN to test two SOTA NER models and two commercial NER APIs,
i.e., Azure NER and AWS NER. We manually verify 784 of the suspicious issues
reported by TIN and find that 702 are erroneous issues, leading to high
precision (85.0%-93.4%) across four categories of NER errors: omission,
over-labeling, incorrect category, and range error. For automated repairing,
TIN achieves a high error reduction rate (26.8%-50.6%) over the four systems
under test, which successfully repairs 1,056 out of the 1,877 reported NER
errors.Comment: Accepted by ESEC/FSE'2
Fairness Testing: A Comprehensive Survey and Analysis of Trends
Unfair behaviors of Machine Learning (ML) software have garnered increasing
attention and concern among software engineers. To tackle this issue, extensive
research has been dedicated to conducting fairness testing of ML software, and
this paper offers a comprehensive survey of existing studies in this field. We
collect 100 papers and organize them based on the testing workflow (i.e., how
to test) and testing components (i.e., what to test). Furthermore, we analyze
the research focus, trends, and promising directions in the realm of fairness
testing. We also identify widely-adopted datasets and open-source tools for
fairness testing
Automating test oracles generation
Software systems play a more and more important role in our everyday life. Many relevant human activities nowadays involve the execution of a piece of software. Software has to be reliable to deliver the expected behavior, and assessing the quality of software is of primary importance to reduce the risk of runtime errors. Software testing is the most common quality assessing technique for software. Testing consists in running the system under test on a finite set of inputs, and checking the correctness of the results. Thoroughly testing a software system is expensive and requires a lot of manual work to define test inputs (stimuli used to trigger different software behaviors) and test oracles (the decision procedures checking the correctness of the results). Researchers have addressed the cost of testing by proposing techniques to automatically generate test inputs. While the generation of test inputs is well supported, there is no way to generate cost-effective test oracles: Existing techniques to produce test oracles are either too expensive to be applied in practice, or produce oracles with limited effectiveness that can only identify blatant failures like system crashes. Our intuition is that cost-effective test oracles can be generated using information produced as a byproduct of the normal development activities. The goal of this thesis is to create test oracles that can detect faults leading to semantic and non-trivial errors, and that are characterized by a reasonable generation cost. We propose two ways to generate test oracles, one derives oracles from the software redundancy and the other from the natural language comments that document the source code of software systems. We present a technique that exploits redundant sequences of method calls encoding the software redundancy to automatically generate test oracles named CCOracles. We describe how CCOracles are automatically generated, deployed, and executed. We prove the effectiveness of CCOracles by measuring their fault-finding effectiveness when combined with both automatically generated and hand-written test inputs. We also present Toradocu, a technique that derives executable specifications from Javadoc comments of Java constructors and methods. From such specifications, Toradocu generates test oracles that are then deployed into existing test suites to assess the outputs of given test inputs. We empirically evaluate Toradocu, showing that Toradocu accurately translates Javadoc comments into procedure specifications. We also show that Toradocu oracles effectively identify semantic faults in the SUT. CCOracles and Toradocu oracles stem from independent information sources and are complementary in the sense that they check different aspects of the system undertest
Improving machine translation systems via isotopic replacement
Machine translation plays an essential role in people’s daily international communication. However, machine translation systems are far from perfect. To tackle this problem, researchers have proposed several approaches to testing machine translation. A promising trend among these approaches is to use word replacement, where only one word in the original sentence is replaced with another word to form a sentence pair. However, precise control of the impact of word replacement remains an outstanding issue in these approaches.
To address this issue, we propose CAT, a novel word-replacement-based approach, whose basic idea is to identify word replacement with controlled impact (referred to as isotopic replacement). To achieve this purpose, we use a neural-based language model to encode the sentence context, and design a neural-network-based algorithm to evaluate context-aware semantic similarity between two words. Furthermore, similar to TransRepair, a state-of-the-art word-replacement-based approach, CAT also provides automatic fixing of revealed bugs without model retraining.
Our evaluation on Google Translate and Transformer indicates that CAT achieves significant improvements over TransRepair. In particular, 1) CAT detects seven more types of bugs than TransRepair; 2) CAT detects 129% more translation bugs than TransRepair; 3) CAT repairs twice more bugs than TransRepair, many of which may bring serious consequences if left unfixed; and 4) CAT has better efficiency than TransRepair in input generation (0.01s v.s. 0.41s) and comparable efficiency with TransRepair in bug repair (1.92s v.s. 1.34s)
Learning to Encode and Classify Test Executions
The challenge of automatically determining the correctness of test executions
is referred to as the test oracle problem and is one of the key remaining
issues for automated testing. The goal in this paper is to solve the test
oracle problem in a way that is general, scalable and accurate.
To achieve this, we use supervised learning over test execution traces. We
label a small fraction of the execution traces with their verdict of pass or
fail. We use the labelled traces to train a neural network (NN) model to learn
to distinguish runtime patterns for passing versus failing executions for a
given program. Our approach for building this NN model involves the following
steps, 1. Instrument the program to record execution traces as sequences of
method invocations and global state, 2. Label a small fraction of the execution
traces with their verdicts, 3. Designing a NN component that embeds information
in execution traces to fixed length vectors, 4. Design a NN model that uses the
trace information for classification, 5. Evaluate the inferred classification
model on unseen execution traces from the program.
We evaluate our approach using case studies from different application
domains: 1. Module from Ethereum Blockchain, 2. Module from PyTorch deep
learning framework, 3. Microsoft SEAL encryption library components, 4. Sed
stream editor, 5. Value pointer library and 6. Nine network protocols from
Linux packet identifier, L7-Filter. We found the classification models for all
subject programs resulted in high precision, recall and specificity, over 95%,
while only training with an average 9% of the total traces. Our experiments
show that the proposed neural network model is highly effective as a test
oracle and is able to learn runtime patterns to distinguish passing and failing
test executions for systems and tests from different application domains
- …