Automatic bug triaging techniques using machine learning and stack traces

When a software system crashes, users have the option to report the crash using automated bug tracking systems. These tools capture software crash and failure data (e.g., stack traces, memory dumps, etc.) from end-users. These data are sent in the form of bug (crash) reports to the software development teams to uncover the causes of the crash and provide adequate fixes. The reports are first assessed (usually in a semi-automatic way) by a group of software analysts, known as triagers. Triagers assign priority to the bugs and redirect them to the software development teams in order to provide fixes. The triaging process, however, is usually very challenging. The problem is that many of these reports are caused by similar faults. Studies have shown that one way to improve the bug triaging process is to detect automatically duplicate (or similar) reports. This way, triagers would not need to spend time on reports caused by faults that have already been handled. Another issue is related to the prioritization of bug reports. Triagers often rely on the information provided by the customers (the report submitters) to prioritize bug reports. However, this task can be quite tedious and requires tool support. Next, triagers route the bug report to the responsible development team based on the subsystem, which caused the crash. Since having knowledge of all the subsystems of an ever-evolving industrial system is impractical, having a tool to automatically identify defective subsystems can significantly reduce the manual bug triaging effort. The main goal of this research is to investigate techniques and tools to help triagers process bug reports. We start by studying the effect of the presence of stack traces in analyzing bug reports. Next, we present a framework to help triagers in each step of the bug triaging process. We propose a new and scalable method to automatically detect duplicate bug reports using stack traces and bug report categorical features. We then propose a novel approach for predicting bug severity using stack traces and categorical features, and finally, we discuss a new method for predicting faulty product and component fields of bug reports. We evaluate the effectiveness of our techniques using bug reports from two large open-source systems. Our results show that stack traces and machine learning methods can be used to automate the bug triaging process, and hence increase the productivity of bug triagers, while reducing costs and efforts associated with manual triaging of bug reports

Improving Bug Triaging Using Software Analytics

RÉSUMÉ La correction de bogues est une activité majeure pendant le développement et maintenance de logiciels. Durant cette activité, le tri de bogues joue un rôle essentiel. Il aide les gestionnaires à allouer leurs ressources limitées et permet aux développeurs de concentrer leurs efforts plus efficacement sur les bogues à haute sévérité. Malheureusement, les techniques du tri de bogues appliquées dans beaucoup d’entreprises ne sont pas toujours efficaces et conduisent à la misclassifications de bogues ou à des retards dans leurs résolutions, qui peuvent mener à la dégradation de la qualité d’un logiciel et à la déception de ses utilisateurs. Une stratégie de tri de bogues améliorée est nécessaire pour aider les gestionnaires à prendre de meilleures décisions, par exemple en accordant des degrés de priorité et sévérité appropriés aux bogues, ce qui permet aux développeurs de corriger les problèmes critiques le plus tôt possible en ignorant les problèmes futiles. Dans ce mémoire, nous utilisons les approches analytiques pour améliorer le tri de bogues. Nous réalisons trois études empiriques. La première étude porte sur la relation entre les corrections de bogues qui ont besoin d’autres corrections ultérieures (corrections supplémentaires) et les bogues qui ont été ouverts plus d’une fois (bogues ré-ouverts). Nous observons que les bogues ré-ouverts occupent entre 21,6% et 33,8% de toutes les corrections supplémentaires. Un grand nombre de bogues ré-ouverts (de 33,0% à 57,5%) n’ont qu’une correction préalable : les bogues originaux ont été fermés prématurément. La deuxième étude concerne les bogues qui provoquent des plantages fréquents, affectant de nombreux utilisateurs. Nous avons observé que ces bogues ne reçoivent pas toujours une attention adéquate même s’ils peuvent sérieusement dégrader la qualité d’un logiciel et même la réputation de l’entreprise. Notre troisième étude concerne les commits qui conduisent à des plantages. Nous avons trouvé que ces commits sont souvent validés par des développeurs moins expérimentés et qu’ils contiennent plus d’additions et de suppressions de lignes de code que les autre commits. Si les entreprises de logiciels pourraient détecter les problèmes susmentionnés pendant la phase du tri de bogues, elles pourraient augmenter l’efficacité de leur correction de bogues et la satisfaction de leurs utilisateurs, réduisant le coût de la maintenance de logiciels. En utilisant plusieurs algorithmes de régression et d’apprentissage automatique, nous avons bâti des modèles statistiques permettant de prédire respectivement des bogues ré-ouverts (avec une précision atteignant 97,0% et un rappel atteignant 65,3%), des bogues affectant un grand nombre d’utilisateurs (avec une précision atteignant 64,2% et un rappel atteignant 98.3%) et des commits induisant des plantages (avec une précision atteignant 61,4% et un rappel atteignant 95,0%). Les entreprises de logiciels peuvent appliquer nos modèles afin d’améliorer leur stratégie de tri de bogues, éviter les misclassifications de bogues et réduire la insatisfaction des utilisateurs due aux plantages.----------ABSTRACT Bug fixing has become a major activity in software development and maintenance. In this process, bug triaging plays an important role. It assists software managers in the allocation of their limited resources and allow developers to focus their efforts more efficiently to solve defects with high severity. Current bug triaging techniques applied in many software organisations may lead to misclassification of bugs, thus delay in bug resolution; resulting in degradation of software quality and users’ frustration. An improved bug triaging strategy would help software managers make better decisions by assigning the right priority and severity to bugs, allowing developers to address critical bugs as soon as possible and ignore the trivial ones. In this thesis, we leverage analytic approaches to conduct three empirical studies aimed at improving bug triaging techniques. The first study investigates the relation between bug fixes that need supplementary fixes and bugs that have been re-opened. We found that re-opened bugs account from 21.6% to 33.8% of all supplementary bug fixes. A considerable number of re-opened bugs (from 33.0% to 57.5%) had only one commit associated: their original bug reports were prematurely closed. The second study focuses on bugs that yield frequent crashes and impact large numbers of users. We found that these bugs were not prioritised by software managers albeit they can seriously decrease user-perceived quality and even the reputation of a software organisation. Our third study examines commits that lead to crashes. We found that these commits are often submitted by less experienced developers and that they contain more addition and deletion of lines of code than other commits. If software organisations can detect the aforementioned problems early on in the bug triaging phase, they can effectively increase their development productivity and users’ satisfaction, while decreasing software maintenance overhead. By using multiple regression and machine learning algorithms, we built statistical models to predict re-opened bugs among bugs that required supplementary bug fixes (with a precision up to 97.0% and a recall up to 65.3%), bugs with high crashing impact (with a precision up to 64.2% and a recall up to 98.3%), and commits inducing future crashes (with a precision up to 61.4% and a recall up to 95.0%). Software organisations can apply our proposed models to improve their bug triaging strategy by assigning bugs to the right developers, avoiding misclassification of bugs, reducing the negative impact of crash-related bugs, and addressing fault-prone code early on before they impact a large user base

Підхід з використанням нечіткої логіки до автоматизованої ідентифікації дефектів в розподілених програмних системах та лінійках програмних продуктів

An approach to the improvement of the efficiency of the bug tracking process in distributed software systems and software product lines via automated identification of duplicate report groups and report groups collected from correlated bugs, combined with bug localization within a software product line is considered. A brief overview of the problem of automated report collection and aggregation is made, several existing software tools and solutions for report management and analysis are reviewed, and basic functionality of a typical report management system is identified. In addition to this, a concept of a report correlation group is introduced and an automated crash report aggregation method based on the rules for comparison of crash signatures, top frames, and frequent closed ordered sub-sets of frames of crash reports is proposed. To evaluate these rules, two separate fuzzy models are built, the first one to calculate the output of the Frequent Closed Ordered Sub-Set Comparison rule, and the second one to interpret and combine the output of all three rules and produce an integrated degree of crash report’s similarity to an existing report correlation group or to another report. A prototype of a report management system with report aggregation capabilities is developed and tested using imported from the publicly available Mozilla Crash Stats project report groups. During the experiment, a precision of 90% and a recall of 81% are achieved. Lastly, an approach to localize the largest identified report groups and represented by them bugs within a concrete software product line based on an information basis consisting of a feature model, a list of software components, and a mapping between features and components is proposed, conclusions are drawn, and goals for the future work are outlined.Розглянуто підхід до підвищення ефективності процесу відстеження помилок в розподілених програмних системах та лінійках програмних продуктів шляхом автоматизованої ідентифікації дубльованих груп звітів та груп звітів, зібраних з корельованих помилок, у поєднанні з локалізацією помилок серед компонентів лінійок програмних продуктів. Зроблено короткий огляд проблеми автоматизованого збору та агрегації звітів, розглянуто кілька існуючих програмних засобів для аналізу звітів, а також визначено основні функціональні можливості типової системи управління звітами. Крім того, запропоновано концепцію кореляційної групи звітів та наведено автоматизований метод агрегації звітів, який базується на правилах порівняння підписів звітів, верхньої форми звітів, та трасувальних стеків звітів про збої. Для оцінки цих правил будуються дві окремі нечіткі моделі – перша для розрахунку результату правила порівняння трасувальних стеків звітів, а друга - для інтерпретації та поєднання результатів усіх трьох правил і створення інтегрованого ступеня подібності звіту про збій з існуючою кореляційною групою звітів або іншим звітом. За допомогою імпорту груп звітів з загальнодоступного репозиторію Mozilla, тестується розроблений прототип системи управління та агрегації звітів. Під час експерименту досягається точність в 90% і повнота в 81%. Нарешті, пропонується підхід до локалізації найбільших ідентифікованих груп звітів та представлених ними помилок у лінійці програмних продуктів на основі інформаційної бази, що складається з функціональної моделі, списку програмних компонентів та взаємозв’язків між функціями та компонентами, робляться висновки та визначаються цілі для подальшої роботи

Bug Triaging with High Confidence Predictions

Correctly assigning bugs to the right developer or team, i.e., bug triaging, is a costly activity. A concerted effort at Ericsson has been done to adopt automated bug triaging to reduce development costs. We also perform a case study on Eclipse bug reports. In this work, we replicate the research approaches that have been widely used in the literature including FixerCache. We apply them on over 10k bug reports for 9 large products at Ericsson and 2 large Eclipse products containing 21 components. We find that a logistic regression classifier including simple textual and categorical attributes of the bug reports has the highest accuracy of 79.00% and 46% on Ericsson and Eclipse bug reports respectively. Ericsson’s bug reports often contain logs that have crash dumps and alarms. We add this information to the bug triage models. We find that this information does not improve the accuracy of bug triaging in Ericsson’s context. Eclipse bug reports contain the stack traces that we add to the bug triaging model. Stack traces are only present in 8% of bug reports and do not improve the triage accuracy. Although our models perform as well as the best ones reported in the literature, a criticism of bug triaging at Ericsson is that accuracy is not sufficient for regular use. We develop a novel approach that only triages bugs when the model has high confidence in the triage prediction. We find that we improve the accuracy to 90% at Ericsson and 70% at Eclipse, but we can make predictions for 62% and 25% of the total Ericsson and Eclipse bug reports,respectively

Improving software engineering processes using machine learning and data mining techniques

The availability of large amounts of data from software development has created an area of research called mining software repositories. Researchers mine data from software repositories both to improve understanding of software development and evolution, and to empirically validate novel ideas and techniques. The large amount of data collected from software processes can then be leveraged for machine learning applications. Indeed, machine learning can have a large impact in software engineering, just like it has had in other fields, supporting developers, and other actors involved in the software development process, in automating or improving parts of their work. The automation can not only make some phases of the development process less tedious or cheaper, but also more efficient and less prone to errors. Moreover, employing machine learning can reduce the complexity of difficult problems, enabling engineers to focus on more interesting problems rather than the basics of development. The aim of this dissertation is to show how the development and the use of machine learning and data mining techniques can support several software engineering phases, ranging from crash handling, to code review, to patch uplifting, to software ecosystem management. To validate our thesis we conducted several studies tackling different problems in an industrial open-source context, focusing on the case of Mozilla

Machine Learning And Deep Learning Based Approaches For Detecting Duplicate Bug Reports With Stack Traces

Many large software systems rely on bug tracking systems to record the submitted bug reports and to track and manage bugs. Handling bug reports is known to be a challenging task, especially in software organizations with a large client base, which tend to receive a considerable large number of bug reports a day. Fortunately, not all reported bugs are new; many are similar or identical to previously reported bugs, also called duplicate bug reports. Automatic detection of duplicate bug reports is an important research topic to help reduce the time and effort spent by triaging and development teams on sorting and fixing bugs. This explains the recent increase in attention to this topic as evidenced by the number of tools and algorithms that have been proposed in academia and industry. The objective is to automatically detect duplicate bug reports as soon as they arrive into the system. To do so, existing techniques rely heavily on the nature of bug report data they operate on. This includes both structural information such as OS, product version, time and date of the crash, and stack traces, as well as unstructured information such as bug report summaries and descriptions written in natural language by end users and developers

Analyze the Performance of Software by Machine Learning Methods for Fault Prediction Techniques

Trend of using the software in daily life is increasing day by day. Software system development is growing more difficult as these technologies are integrated into daily life. Therefore, creating highly effective software is a significant difficulty. The quality of any software system continues to be the most important element among all the required characteristics. Nearly one-third of the total cost of software development goes toward testing. Therefore, it is always advantageous to find a software bug early in the software development process because if it is not found early, it will drive up the cost of the software development. This type of issue is intended to be resolved via software fault prediction. There is always a need for a better and enhanced prediction model in order to forecast the fault before the real testing and so reduce the flaws in the time and expense of software projects. The various machine learning techniques for classifying software bugs are discussed in this paper