5 research outputs found
What Causes My Test Alarm? Automatic Cause Analysis for Test Alarms in System and Integration Testing
Driven by new software development processes and testing in clouds, system
and integration testing nowadays tends to produce enormous number of alarms.
Such test alarms lay an almost unbearable burden on software testing engineers
who have to manually analyze the causes of these alarms. The causes are
critical because they decide which stakeholders are responsible to fix the bugs
detected during the testing. In this paper, we present a novel approach that
aims to relieve the burden by automating the procedure. Our approach, called
Cause Analysis Model, exploits information retrieval techniques to efficiently
infer test alarm causes based on test logs. We have developed a prototype and
evaluated our tool on two industrial datasets with more than 14,000 test
alarms. Experiments on the two datasets show that our tool achieves an accuracy
of 58.3% and 65.8%, respectively, which outperforms the baseline algorithms by
up to 13.3%. Our algorithm is also extremely efficient, spending about 0.1s per
cause analysis. Due to the attractive experimental results, our industrial
partner, a leading information and communication technology company in the
world, has deployed the tool and it achieves an average accuracy of 72% after
two months of running, nearly three times more accurate than a previous
strategy based on regular expressions.Comment: 12 page
An experimental analysis of assessor specific bias in a programming assessment in multi-assessor scenarios utilizing an eye tracker
It has been experienced and reported by academic institutions around the globe that marking of most subject’s assessment scripts vary when different assessors are utilized for a given subject. To understand the difference, we capture and analysis pattern while they are marking the scripts. For this, a Java programming assessment from a real life university examination is marked by independent assessors. The assessors marked the scanned assessment scripts on a computer screen in front of an Eye tracker machine and their eye gaze data were recorded real time. Data indicate that different assessors marked the same answer script differently and their visual pattern are also varied although they were given the exact same instructions which demonstrates bias to a degree. For quality marking, several findings including the number of assessors needed are also presented in this manuscript