Spectrum-Based Fault Localization in Model Transformations

Model transformations play a cornerstone role in Model-Driven Engineering (MDE), as they provide the essential mechanisms for manipulating and transforming models. The correctness of software built using MDE techniques greatly relies on the correctness of model transformations. However, it is challenging and error prone to debug them, and the situation gets more critical as the size and complexity of model transformations grow, where manual debugging is no longer possible. Spectrum-Based Fault Localization (SBFL) uses the results of test cases and their corresponding code coverage information to estimate the likelihood of each program component (e.g., statements) of being faulty. In this article we present an approach to apply SBFL for locating the faulty rules in model transformations. We evaluate the feasibility and accuracy of the approach by comparing the effectiveness of 18 different stateof- the-art SBFL techniques at locating faults in model transformations. Evaluation results revealed that the best techniques, namely Kulcynski2, Mountford, Ochiai, and Zoltar, lead the debugger to inspect a maximum of three rules to locate the bug in around 74% of the cases. Furthermore, we compare our approach with a static approach for fault localization in model transformations, observing a clear superiority of the proposed SBFL-based method.Comisión Interministerial de Ciencia y Tecnología TIN2015-70560-RJunta de Andalucía P12-TIC-186

Evaluating the Impact of Experimental Assumptions in Automated Fault Localization

peer reviewe

Integration of an Automatic Fault Localization Tool in an IDE and its Evaluation

Debugging is one of the most demanding and error-prone tasks in software development. Trying to address bugs has become overall more expensive as the software complexity and size have increased. As a result, several researchers attempted to improve the developers’ debugging experience and efficiency by automating as much of the process as possible. Existing auto-finding tools will assist developers in automatically detecting bugs, however, they are not yet widely available to software engineers. Making such tools available to developers can save debugging time and increase the productivity. Subsequently, the main goal of this dissertation is to incorporate an automatic fault localization tool into an Integrated Development Environment (IDE). The selected IDE was Visual Studio Code, a source-code editor developed by Microsoft for Windows, Linux, and macOS. Visual Studio Code is one of the most used IDEs and is known for its flexible API, which allows nearly every aspect of it to be customized. Furthermore, the chosen automatic fault localization tool was FLACOCO, a recent fault localization tool for Java that supports up to the most recent versions. Nonetheless, this document contains a full overview of several fault localization methodologies and tools, as well as an explanation of the complete planning and development process of the produced Visual Studio Code extension. After the development and deployment were completed, an evaluation was carried out. The extension was evaluated through a user study in which thirty Java professionals took part. The test had two parts: the first involved users using the extension to complete two debugging tasks in previously unknown projects, and the second had them filling out a satisfaction questionnaire for further analysis. Finally, the results show that the extension was a success, with the system being rated positively in all areas. However, it may be revised in light of the questionnaire responses, with the suggestions received being considered for future work.A depuração é uma das tarefas mais exigentes e propensas a erros no desenvolvimento de software. Tentar resolver esses erros tornou-se mais dispendioso com os incrementos de complexidade e tamanho do software. Deste modo, ao longo dos últimos anos, vários investigadores tentaram melhorar a experiência de depuração e a eficiência dos desenvolvedores automatizando o máximo possível do processo. Existem ferramentas de localização de defeitos que assistem os desenvolvedores na detecção automática de bugs, no entanto estas ainda não se encontram amplamente disponíveis para os programadores. Tornar essas ferramentas disponíveis para todos certamente iria resultar na redução do tempo de depuração e no aumento da produtividade. Assim sendo, o principal objetivo desta dissertação é incorporar uma ferramenta de localização automática de defeitos num IDE. Em termos de IDE, o Visual Studio Code, um editor de código-fonte desenvolvido pela Microsoft para Windows, Linux e macOS, foi selecionado. Este IDE tem ganho bastante popularidade, sendo um dos IDEs mais utilizados mundialmente. Além disso, o Visual Studio Code é reconhecido pela sua API flexível, que permite que quase todos os seus aspectos sejam personalizados. Adicionalmente, o FLACOCO, uma ferramenta de localização de defeitos baseada em SFL que suporta até as versões mais recentes do Java, foi escolhida como ferramenta de localização automática de defeitos. Além do mais, esta dissertação contém um estudo sobre as técnicas de localização automática de defeitos e as suas ferramentas, bem como uma explicação do planeamento e implementação da extensão criada para o Visual Studio Code. Após o término da implementação e a posterior implantação, foi efetuada a sua avaliação. Procedeu-se a um teste de utilização com a participação de treze utilizadores proficientes na linguagem Java. O teste foi composto por duas componentes: na primeira os utilizadores utilizaram a extensão para completar duas tarefas de depuração em projetos por eles desconhecidos e na segunda foi-lhes fornecido um questionário de satisfação para posterior análise. Os resultados obtidos sugerem que a extensão foi um sucesso, sendo que o sistema foi positivamente avaliado em todos os aspetos. No entanto a mesma poderá ser aprimorada tendo em consideração o feedback obtido na secção de resposta livre do questionário, sendo que o mesmo foi bastante valioso e as sugestões apuradas vieram a ser consideradas para trabalho futuro

Back to the Future! Studying Data Cleanness in Defects4J and its Impact on Fault Localization

For software testing research, Defects4J stands out as the primary benchmark dataset, offering a controlled environment to study real bugs from prominent open-source systems. However, prior research indicates that Defects4J might include tests added post-bug report, embedding developer knowledge and affecting fault localization efficacy. In this paper, we examine Defects4J's fault-triggering tests, emphasizing the implications of developer knowledge of SBFL techniques. We study the timelines of changes made to these tests concerning bug report creation. Then, we study the effectiveness of SBFL techniques without developer knowledge in the tests. We found that 1) 55% of the fault-triggering tests were newly added to replicate the bug or to test for regression; 2) 22% of the fault-triggering tests were modified after the bug reports were created, containing developer knowledge of the bug; 3) developers often modify the tests to include new assertions or change the test code to reflect the changes in the source code; and 4) the performance of SBFL techniques degrades significantly (up to --415% for Mean First Rank) when evaluated on the bugs without developer knowledge. We provide a dataset of bugs without developer insights, aiding future SBFL evaluations in Defects4J and informing considerations for future bug benchmarks