A preliminary evaluation of text-based and dependency-based techniques for determining the origins of bugs

A crucial step in understanding the life cycle of software bugs is identifying their origin. Unfortunately this information is not usually recorded and recovering it at a later date is challenging. Recently two approaches have been developed that attempt to solve this problem: the text approach and the dependency approach. However only limited evaluation has been carried out on their effectiveness so far, partially due to the lack of data sets linking bugs to their introduction. Producing such data sets is both time-consuming and challenging due to the subjective nature of the problem. To improve this, the origins of 166 bugs in two open-source projects were manually identified. These were then compared to a simulation of the approaches. The results show that both approaches were partially successful across a variety of different types of bugs. They achieved a precision of 29%{79% and a recall of 40%{70%, and could perform better when combined. However there remain a number of challenges to overcome in future development|large commits, unrelated changes and large numbers of versions between the origin and the x all reduce their effectiveness

Bugs & fixes in Eclipse & Rachota

<p>Bugs & fixes in Eclipse & Rachota, as used in S. Davies, M. Roper and M.Wood, "A preliminary evaluation of text-based and dependency-based techniques for determining the origins of bugs", in Proc. WCRE, 2011 (http://dx.doi.org/10.1109/WCRE.2011.32)</p> <p> </p> <p>* Fields are comma-separated and unquoted</p> <p>* Fields which contain multiple values are pipe-separated</p> <p>* ? means no result for that field, and can be interpreted slightly differently for each field:</p> <p>** Origin(s): No origin found. The reason why is recorded in the Origin Class field</p> <p>** TA Result(s),DA Result(s): The approach would not return any result, or the Origin Class was Unclear</p> <p>** TA Most Frequent, TA Most Recent: The result of the variation was not recorded, either because it was obvious (in the case of the original approach returning 0 or 1 results) or immaterial (in the case where all results of the original approach are wrong)</p