2 research outputs found
What Causes My Test Alarm? Automatic Cause Analysis for Test Alarms in System and Integration Testing
Driven by new software development processes and testing in clouds, system
and integration testing nowadays tends to produce enormous number of alarms.
Such test alarms lay an almost unbearable burden on software testing engineers
who have to manually analyze the causes of these alarms. The causes are
critical because they decide which stakeholders are responsible to fix the bugs
detected during the testing. In this paper, we present a novel approach that
aims to relieve the burden by automating the procedure. Our approach, called
Cause Analysis Model, exploits information retrieval techniques to efficiently
infer test alarm causes based on test logs. We have developed a prototype and
evaluated our tool on two industrial datasets with more than 14,000 test
alarms. Experiments on the two datasets show that our tool achieves an accuracy
of 58.3% and 65.8%, respectively, which outperforms the baseline algorithms by
up to 13.3%. Our algorithm is also extremely efficient, spending about 0.1s per
cause analysis. Due to the attractive experimental results, our industrial
partner, a leading information and communication technology company in the
world, has deployed the tool and it achieves an average accuracy of 72% after
two months of running, nearly three times more accurate than a previous
strategy based on regular expressions.Comment: 12 page
Improving Salience Retention and Identification in the Automated Filtering of Event Log Messages
Event log messages are currently the only genuine interface through which computer systems
administrators can effectively monitor their systems and assemble a mental perception
of system state. The popularisation of the Internet and the accompanying meteoric
growth of business-critical systems has resulted in an overwhelming volume of event log
messages, channeled through mechanisms whose designers could not have envisaged the
scale of the problem. Messages regarding intrusion detection, hardware status, operating
system status changes, database tablespaces, and so on, are being produced at the rate
of many gigabytes per day for a significant computing environment.
Filtering technologies have not been able to keep up. Most messages go unnoticed; no
filtering whatsoever is performed on them, at least in part due to the difficulty of implementing
and maintaining an effective filtering solution. The most commonly-deployed
filtering alternatives rely on regular expressions to match pre-defi ned strings, with 100%
accuracy, which can then become ineffective as the code base for the software producing
the messages 'drifts' away from those strings. The exactness requirement means all possible
failure scenarios must be accurately anticipated and their events catered for with
regular expressions, in order to make full use of this technique.
Alternatives to regular expressions remain largely academic. Data mining, automated
corpus construction, and neural networks, to name the highest-profi le ones, only produce
probabilistic results and are either difficult or impossible to alter in any deterministic way.
Policies are therefore not supported under these alternatives.
This thesis explores a new architecture which utilises rich metadata in order to avoid the
burden of message interpretation. The metadata itself is based on an intention to improve
end-to-end communication and reduce ambiguity. A simple yet effective filtering scheme
is also presented which fi lters log messages through a short and easily-customisable set
of rules. With such an architecture, it is envisaged that systems administrators could
signi ficantly improve their awareness of their systems while avoiding many of the false-positives
and -negatives which plague today's fi ltering solutions