137 research outputs found
Amortising the Cost of Mutation Based Fault Localisation using Statistical Inference
Mutation analysis can effectively capture the dependency between source code
and test results. This has been exploited by Mutation Based Fault Localisation
(MBFL) techniques. However, MBFL techniques suffer from the need to expend the
high cost of mutation analysis after the observation of failures, which may
present a challenge for its practical adoption. We introduce SIMFL (Statistical
Inference for Mutation-based Fault Localisation), an MBFL technique that allows
users to perform the mutation analysis in advance against an earlier version of
the system. SIMFL uses mutants as artificial faults and aims to learn the
failure patterns among test cases against different locations of mutations.
Once a failure is observed, SIMFL requires either almost no or very small
additional cost for analysis, depending on the used inference model. An
empirical evaluation of SIMFL using 355 faults in Defects4J shows that SIMFL
can successfully localise up to 103 faults at the top, and 152 faults within
the top five, on par with state-of-the-art alternatives. The cost of mutation
analysis can be further reduced by mutation sampling: SIMFL retains over 80% of
its localisation accuracy at the top rank when using only 10% of generated
mutants, compared to results obtained without sampling
J-model: an open and social ensemble learning architecture for classification
Ensemble learning is a promising direction of research in machine learning, in which an ensemble
classifier gives better predictive and more robust performance for classification problems
by combining other learners. Meanwhile agent-based systems provide frameworks to
share knowledge from multiple agents in an open context. This thesis combines multi-agent
knowledge sharing with ensemble methods to produce a new style of learning system for open
environments.
We now are surrounded by many smart objects such as wireless sensors, ambient communication
devices, mobile medical devices and even information supplied via other humans. When
we coordinate smart objects properly, we can produce a form of collective intelligence from
their collaboration. Traditional ensemble methods and agent-based systems have complementary
advantages and disadvantages in this context. Traditional ensemble methods show better
classification performance, while agent-based systems might not guarantee their performance
for classification. Traditional ensemble methods work as closed and centralised systems
(so they cannot handle classifiers in an open context), while agent-based systems are natural
vehicles for classifiers in an open context.
We designed an open and social ensemble learning architecture, named J-model, to merge the
conflicting benefits of the two research domains. The J-model architecture is based on a service
choreography approach for coordinating classifiers. Coordination protocols are defined by
interaction models that describe how classifiers will interact with one another in a peer-to-peer
manner. The peer ranking algorithm recommends more appropriate classifiers to participate in
an interaction model to boost the success rate of results of their interactions. Coordinated participant
classifiers who are recommended by the peer ranking algorithm become an ensemble
classifier within J-model.
We evaluated J-model’s classification performance with 13 UCI machine learning benchmark
data sets and a virtual screening problem as a realistic classification problem. J-model showed
better performance of accuracy, for 9 benchmark sets out of 13 data sets, than 8 other representative
traditional ensemble methods. J-model gave better results of specificity for 7 benchmark
sets. In the virtual screening problem, J-model gave better results for 12 out of 16 bioassays
than already published results. We defined different interaction models for each specific classification
task and the peer ranking algorithm was used across all the interaction models.
Our research contributions to knowledge are as follows. First, we showed that service choreography
can be an effective ensemble coordination method for classifiers in an open context. Second, we used interaction models that implement task specific coordinations of classifiers to
solve a variety of representative classification problems. Third, we designed the peer ranking
algorithm which is generally and independently applicable to the task of recommending appropriate
member classifiers from a classifier pool based on an open pool of interaction models
and classifiers
Guiding Deep Learning System Testing using Surprise Adequacy
Deep Learning (DL) systems are rapidly being adopted in safety and security
critical domains, urgently calling for ways to test their correctness and
robustness. Testing of DL systems has traditionally relied on manual collection
and labelling of data. Recently, a number of coverage criteria based on neuron
activation values have been proposed. These criteria essentially count the
number of neurons whose activation during the execution of a DL system
satisfied certain properties, such as being above predefined thresholds.
However, existing coverage criteria are not sufficiently fine grained to
capture subtle behaviours exhibited by DL systems. Moreover, evaluations have
focused on showing correlation between adversarial examples and proposed
criteria rather than evaluating and guiding their use for actual testing of DL
systems. We propose a novel test adequacy criterion for testing of DL systems,
called Surprise Adequacy for Deep Learning Systems (SADL), which is based on
the behaviour of DL systems with respect to their training data. We measure the
surprise of an input as the difference in DL system's behaviour between the
input and the training data (i.e., what was learnt during training), and
subsequently develop this as an adequacy criterion: a good test input should be
sufficiently but not overtly surprising compared to training data. Empirical
evaluation using a range of DL systems from simple image classifiers to
autonomous driving car platforms shows that systematic sampling of inputs based
on their surprise can improve classification accuracy of DL systems against
adversarial examples by up to 77.5% via retraining
Reducing DNN labelling cost using surprise adequacy: An industrial case study for autonomous driving
Deep Neural Networks (DNNs) are rapidly being adopted by the automotive industry, due to their impressive performance in tasks that are essential for autonomous driving. Object segmentation is one such task: its aim is to precisely locate boundaries of objects and classify the identified objects, helping autonomous cars to recognise the road environment and the traffic situation. Not only is this task safety critical, but developing a DNN based object segmentation module presents a set of challenges that are significantly different from traditional development of safety critical software. The development process in use consists of multiple iterations of data collection, labelling, training, and evaluation. Among these stages, training and evaluation are computation intensive while data collection and labelling are manual labour intensive. This paper shows how development of DNN based object segmentation can be improved by exploiting the correlation between Surprise Adequacy (SA) and model performance. The correlation allows us to predict model performance for inputs without manually labelling them. This, in turn, enables understanding of model performance, more guided data collection, and informed decisions about further training. In our industrial case study the technique allows cost savings of up to 50% with negligible evaluation inaccuracy. Furthermore, engineers can trade off cost savings versus the tolerable level of inaccuracy depending on different development phases and scenarios
Learning Test-Mutant Relationship for Accurate Fault Localisation
Context: Automated fault localisation aims to assist developers in the task
of identifying the root cause of the fault by narrowing down the space of
likely fault locations. Simulating variants of the faulty program called
mutants, several Mutation Based Fault Localisation (MBFL) techniques have been
proposed to automatically locate faults. Despite their success, existing MBFL
techniques suffer from the cost of performing mutation analysis after the fault
is observed. Method: To overcome this shortcoming, we propose a new MBFL
technique named SIMFL (Statistical Inference for Mutation-based Fault
Localisation). SIMFL localises faults based on the past results of mutation
analysis that has been done on the earlier version in the project history,
allowing developers to make predictions on the location of incoming faults in a
just-in-time manner. Using several statistical inference methods, SIMFL models
the relationship between test results of the mutants and their locations, and
subsequently infers the location of the current faults. Results: The empirical
study on Defects4J dataset shows that SIMFL can localise 113 faults on the
first rank out of 224 faults, outperforming other MBFL techniques. Even when
SIMFL is trained on the predicted kill matrix, SIMFL can still localise 95
faults on the first rank out of 194 faults. Moreover, removing redundant
mutants significantly improves the localisation accuracy of SIMFL by the number
of faults localised at the first rank up to 51. Conclusion: This paper proposes
a new MBFL technique called SIMFL, which exploits ahead-of-time mutation
analysis to localise current faults. SIMFL is not only cost-effective, as it
does not need a mutation analysis after the fault is observed, but also capable
of localising faults accurately.Comment: Paper accepted for publication at IST. arXiv admin note: substantial
text overlap with arXiv:1902.0972
Reducing DNN Labelling Cost using Surprise Adequacy: An Industrial Case Study for Autonomous Driving
Deep Neural Networks (DNNs) are rapidly being adopted by the automotive
industry, due to their impressive performance in tasks that are essential for
autonomous driving. Object segmentation is one such task: its aim is to
precisely locate boundaries of objects and classify the identified objects,
helping autonomous cars to recognise the road environment and the traffic
situation. Not only is this task safety critical, but developing a DNN based
object segmentation module presents a set of challenges that are significantly
different from traditional development of safety critical software. The
development process in use consists of multiple iterations of data collection,
labelling, training, and evaluation. Among these stages, training and
evaluation are computation intensive while data collection and labelling are
manual labour intensive. This paper shows how development of DNN based object
segmentation can be improved by exploiting the correlation between Surprise
Adequacy (SA) and model performance. The correlation allows us to predict model
performance for inputs without manually labelling them. This, in turn, enables
understanding of model performance, more guided data collection, and informed
decisions about further training. In our industrial case study the technique
allows cost savings of up to 50% with negligible evaluation inaccuracy.
Furthermore, engineers can trade off cost savings versus the tolerable level of
inaccuracy depending on different development phases and scenarios.Comment: to be published in Proceedings of the 28th ACM Joint European
Software Engineering Conference and Symposium on the Foundations of Software
Engineerin
Bridging the Divide: Unraveling the Knowledge Gap in Data Visualization Research and Practice
Empirical research on perception and cognition has laid the foundation for
visualization design, often yielding useful design guidelines for
practitioners. However, it remains uncertain how well practitioners stay
informed about such crucial visualization design knowledge. In this paper, we
employed a mixed-method approach to explore the knowledge gap between
visualization research and real-world design guidelines. We initially collected
existing design guidelines from various sources and empirical studies from
diverse publishing venues, analyzing their alignment and uncovering missing
links and inconsistent knowledge. Subsequently, we conducted surveys and
interviews with practitioners and researchers to gain further insights into
their experiences and attitudes towards design guidelines and empirical
studies, and their views on the knowledge gap between research and practice.
Our findings highlight the similarities and differences in their perspectives
and propose strategies to bridge the divide in visualization design knowledge.Comment: 15 pages, 5 figure
Fabrication of highly ordered multilayer thin films and its applications
A new method is introduced to build up organic/organic multilayer films composed of cationic poly(allylamine hydrochloride) (PAH) and negatively charged poly(sodium 4-styrenesulfonate) (PSS) by using the spinning process. The adsorption process is governed by both the viscous force induced by fast solvent elimination and the electrostatic interaction between oppositely charged species. On the other hand, the centrifugal and air shear forces applied by the spinning process significantly enhance the desorption of weakly bound polyelectrolyte chains and also induce the planarization of the adsorbed polyelectrolyte layer. The film thickness per bilayer adsorbed by the conventional dipping process and the spinning process was found to be about 4 å and 24 å, respectively. The surface of the multilayer films prepared with the spinning process is quite homogeneous and smooth. Electroluminescence (EL) devices composed of alternating poly(p-phenylene vinylene) (PPV) and polyanions films show higher quantum efficiency when prepared by the spin self-assembly (SA) method.This work was financially supported by the National Research
Laboratory Program (Grant M1-0104-00-0191) and funded in part
by the Ministry of Education through the Brain Korea 21 Program
at Seoul National University
Development of a Remote Testing System for Performance of Gas Leakage Detectors
In this research, we designed a remote system to test parameters of gas detectors such as gas concentration and initial response time. This testing system is available to measure two gas instruments simultaneously. First of all, we assembled an experimental jig with a square structure. Those parts are included with a glass flask, two high-quality cameras, and two Ethernet modems for transmitting data. This remote gas detector testing system extracts numerals from videos with continually various gas concentrations while LCDs show photographs from cameras. Extracted numeral data are received to a laptop computer through Ethernet modem. And then, the numerical data with gas concentrations and the measured initial response speeds are recorded and graphed. Our remote testing system will be diversely applied on gas detector’s test and will be certificated in domestic and international countries
Spin-coated ultrathin multilayers and their micropatterning using microfluidic channels
A new method is introduced to build up organic/organic multilayer films composed of cationic poly(allylamine
hydrochloride) (PAH) and negatively charged poly (sodium 4-styrenesulfonate) (PSS) using the spinning
process. The adsorption process is governed by both the viscous force induced by fast solvent
elimination and the electrostatic interaction between oppositely charged species. On the other hand, the centrifugal
and air shear forces applied by the spinning process significantly enhances desorption of weakly
bound polyelectrolyte chains and also induce the planarization of the adsorbed polyelectrolyte layer. The
film thickness per bilayer adsorbed by the conventional dipping process and the spinning process was found
to be about 4 Å and 24 Å, respectively. The surface of the multilayer films prepared with the spinning process
is quite homogeneous and smooth. Also, a new approach to create multilayer ultrathin films with welldefined
micropatterns in a short process time is introduced. To achieve such micropatterns with high line
resolution in organic multilayer films, microfluidic channels were combined with the convective self-assembly
process employing both hydrogen bonding and electrostatic intermolecular interactions. The channels
were initially filled with polymer solution by capillary pressure and the residual solution was then removed
by the spinning process.This work was financially supported by the National
Research Laboratory Program (Grant M1-0104-00-0191)
and funded in part by the Ministry of Education through
the Brain Korea 21 Program at Seoul National University
- …