Automated software issue triage in large scale industrial context

Software issue reports are the documents describing the problems users face when using a software product and software issue triage is the process of validating and assigning these issue reports. In practice, issue triage is carried out manually by experts or developers. In large scale industrial contexts, hundreds of software products exist and hundreds of issue reports are filed every day. It takes a great amount of human effort to triage these reports and failure to solve them on time results in customer dissatisfaction. In this thesis, we automate the issue triage process by using data mining approaches and share our experience gained by deploying the resulting system in a large scale industrial setting. Deployment of such a system presented us not only with an opportunity to observe the practical effects of automation, but also to carry out user studies, both of which have not been done before in this context. Furthermore, we developed and empirically evaluated methods on how to create human-readable, non-technical explanations for the predictions made, and on how to monitor and detect deteriorations in accuracies in an online manner. In our efforts to improve the performance, we analyzed the incorrectly assigned issue reports. We realized that many of them have attachments with them, which are mostly screenshots, and such reports generally have short or insufficient descriptions for the problem. Based on these observations, we further carried out studies on how to ensure that we detect the missing information in the descriptions of issue reports automatically and how we can use the attached screenshots as an additional source of information, in order to improve the performance of the automation

An exploratory study on improving automated issue triage with attached screenshots

Issue triage is a manual and time consuming process for both openand closed source software projects. Triagers first validate the issuereports and then find the appropriate developers or teams to solvethem. In our industrial case, we automated the assignment part ofthe problem with a machine learning based approach. However,the automated system's average accuracy performance is 3% belowthe human triagers' performance. In our effort to improve ourapproach, we analyzed the incorrectly assigned issue reports andrealized that many of them have attachments with them, whichare mostly screenshots. Such issue reports generally have shortdescriptions compared to the ones without attachments, which weconsider as one of the reasons for incorrect classification. In thisstudy, we describe our proposed approach to include this new pieceof information for issue triage and present the initial results

Automated issue assignment: results and insights from an industrial case

We automate the process of assigning issue reports to development teams by using data mining approaches and share our experience gained by deploying the resulting system, calledIssueTAG, atSofttech. Being a subsidiary of the largest private bank in Turkey, Softtech on average receives 350 issue reports daily from the field, which need to be handled with utmost importance and urgency. IssueTAG has been making all the issue assignments at Softtech since its deployment on Jan 12, 2018. Deploying IssueTAG presented us not only with an unprecedented opportunity to observe the practical effects of automated issue assignment, but also with an opportunity to carry out user studies, both of which (to the best of our knowledge) have not been done before in this context. We first empirically determine the data mining approach to be used in IssueTAG. We then deploy IssueTAG and make a number of valuable observations. First, it is not just about deploying a system for automated issue assignment, but also about designing/changing the assignment process around the system. Second, the accuracy of the assignments does not have to be higher than that of manual assignments in order for the system to be useful. Third, deploying such a system requires the development of additional functionalities, such as creating human-readable explanations for the assignments and detecting deteriorations in assignment accuracies, for both of which we have developed and empirically evaluated different approaches. Last but not least, stakeholders do not necessarily resist change and gradual transition helps build confidence

Inferring dependencies among web services with predictive and statistical analysis of system logs

Software system behaviour analysis is a challenging research problem in software engineering. The main reason for this is the lack of real data from large industrial systems. Softtech Inc. is a subsidiary of a large private bank in Turkey and this study is aimed to analyse mapping the services architecture and the software system health of a particular department at Softtech using specific software web service logs. The services that are the subject of this study consist of 196 web services related to credit and credit card application transactions from various channels. While these processes are related to similar applications, they call various web services that perform different operations in the background. Related services account for 2 million logs daily. We have conducted empirical and statistical analysis on the data, in order to infer the correlations and dependencies among the observed web services. Hypothetically, we think there are 3 types of dependencies between the web services. In our experiments, we used average response times and the number of times web services are called at specific time intervals as input data. The results suggest that they can be used for inferring that there is a dependency between two web services. In this preliminary work for dependency inference from unstructured web services' log data, we have utilized simple statistical analysis tools to derive important insight about the collection of services under our observation. The results have encouraged us to carry on with a more detailed analysis approach to further advance our research efforts

Improvement of software issue record management process in an industrial context (Endüstriyel bağlamda yazılım olay kaydı yönetim sürecinin iyileştirilmesi)

Issue record management systems are used for tracking software problems. Through these systems, reporting these problems becomes easier and more problems can be detected and solved, which in turn increases the quality of the software products and customer satisfaction with faster solutions. However, as the number of bug reports in the system increases, it becomes more difficult to validate the records, prioritize them and assign them to the relevant software teams. Softtech Inc. is a software company that serves in the banking sector in Turkey and with more than 300 issue records submitted daily, has to manage this process and improve continuously. For this purpose, “application support team” has been set up to examine and solve issue records and take over other operational tasks from software developers. In this study, the issue record solution process is examined with "value stream mapping" methodology, with the aim of improving by measuring the related data is analyzed, improvements achieved and other research studies planned to be carried out in the forthcoming period are shared