A Systematic Mapping Study of Code Quality in Education -- with Complete Bibliography

While functionality and correctness of code has traditionally been the main focus of computing educators, quality aspects of code are getting increasingly more attention. High-quality code contributes to the maintainability of software systems, and should therefore be a central aspect of computing education. We have conducted a systematic mapping study to give a broad overview of the research conducted in the field of code quality in an educational context. The study investigates paper characteristics, topics, research methods, and the targeted programming languages. We found 195 publications (1976-2022) on the topic in multiple databases, which we systematically coded to answer the research questions. This paper reports on the results and identifies developments, trends, and new opportunities for research in the field of code quality in computing education

Mining Fix Patterns for FindBugs Violations

In this paper, we first collect and track a large number of fixed and unfixed violations across revisions of software. The empirical analyses reveal that there are discrepancies in the distributions of violations that are detected and those that are fixed, in terms of occurrences, spread and categories, which can provide insights into prioritizing violations. To automatically identify patterns in violations and their fixes, we propose an approach that utilizes convolutional neural networks to learn features and clustering to regroup similar instances. We then evaluate the usefulness of the identified fix patterns by applying them to unfixed violations. The results show that developers will accept and merge a majority (69/116) of fixes generated from the inferred fix patterns. It is also noteworthy that the yielded patterns are applicable to four real bugs in the Defects4J major benchmark for software testing and automated repair.Comment: Accepted for IEEE Transactions on Software Engineerin

A Systematic Mapping Study of Code Quality in Education -- with Complete Bibliography

While functionality and correctness of code has traditionally been the main focus of computing educators, quality aspects of code are getting increasingly more attention. High-quality code contributes to the maintainability of software systems, and should therefore be a central aspect of computing education. We have conducted a systematic mapping study to give a broad overview of the research conducted in the field of code quality in an educational context. The study investigates paper characteristics, topics, research methods, and the targeted programming languages. We found 195 publications (1976-2022) on the topic in multiple databases, which we systematically coded to answer the research questions. This paper reports on the results and identifies developments, trends, and new opportunities for research in the field of code quality in computing education

Introductory programming: a systematic literature review

As computing becomes a mainstream discipline embedded in the school curriculum and acts as an enabler for an increasing range of academic disciplines in higher education, the literature on introductory programming is growing. Although there have been several reviews that focus on specific aspects of introductory programming, there has been no broad overview of the literature exploring recent trends across the breadth of introductory programming. This paper is the report of an ITiCSE working group that conducted a systematic review in order to gain an overview of the introductory programming literature. Partitioning the literature into papers addressing the student, teaching, the curriculum, and assessment, we explore trends, highlight advances in knowledge over the past 15 years, and indicate possible directions for future research

Um metamodelo para apoiar a formalização de convenções de codificação

Orientador: Leonardo MontecchiDissertação (mestrado) - Universidade Estadual de Campinas, Instituto de ComputaçãoResumo: As convenções de codificação são um meio de melhorar a confiabilidade dos sistemas de software. Elas podem ser estabelecidas por vários motivos, desde melhorar a legibilidade do código até evitar a introdução de falhas de segurança. No entanto, convenções de codificação geralmente vêm na forma de documentos textuais em linguagem natural, o que as torna difíceis de gerenciar e aplicar. Seguindo os princípios de engenharia orientados a modelos, nesta dissertação, propomos uma abordagem e uma linguagem para especificar convenções de codificação usando modelos estruturados. Chamamos tal linguagem de Coding Conventions Specification Language (CCSL). Também propomos uma transformação de modelo para gerar automaticamente verificadores a partir de uma especificação CCSL para encontrar violações das regras especificadas. Para avaliar a proposta, realizamos dois experimentos. O primeiro experimento tem como objetivo avaliar o metamodelo CCSL, enquanto o outro tem como objetivo verificar a capacidade dos verificadores de encontrar violações da regra especificada nos códigos Java. Os resultados obtidos são promissores e sugerem que a abordagem proposta é viável. No entanto, eles também destacam que muitos desafios ainda precisam ser superados. No primeiro experimento, analisamos um total de 216 regras individuais de dois grandes conjuntos de convenções de codificação existentes. No geral, foi possível representar 63% das regras de codificação consideradas usando nossa linguagem. No segundo experimento, selecionamos 53 regras dentre as implementadas na ferramenta PMD (um analisador de código popular) para comparar os resultados entre nossa ferramenta e a ferramenta PMD em três projetos reais. Em geral, alcançamos resultados iguais ou melhores da ferramenta PMD em mais da metade das regras selecionadas (79%), enquanto apenas 6% das regras não puderam ser especificadas usando nossa linguagem. Nas regras restantes, os resultados apresentados foram diferentes para cada uma das ferramentas. Concluímos discutindo instruções para trabalhos futurosAbstract: Coding conventions are a means to improve the reliability of software systems. They can be established for many reasons, ranging from improving the readability of code to avoiding the introduction of security flaws. However, coding conventions often come in the form of textual documents in natural language, which makes them hard to be managed and to enforced. Following model-driven engineering principles, in this dissertation we propose an approach and language for specifying coding conventions using structured models. We call this language Coding Conventions Specification Language (CCSL). We also propose a model transformation to concretely generate checkers to find violations of the rules specified with our language. To evaluate the proposal, we performed two experiments. The first experiment aims to evaluate the Coding Conventions Specification Language metamodel, while the other aims to check the capability of the derived checkers to find violations of the specified rule in Java codes. The obtained results are promising and suggest that the proposed approach is feasible. However, they also highlight that many challenges still need to be overcome. In the first experiment, we analyzed a total of 216 individual rules from two large sets of existing coding conventions. Overall, it was possible to represent 63% of the considered coding rules using our language. In the second experiment, we selected 53 rules from those implemented in the PMD tool to compare the results between our tool and the PMD tool in three real projects. In general, we achieve equal or better results of the PMD tool in more than half of the selected rules (79%), while only 6% of the rules could not be specified using our language. There are also cases where PMD performed better than our approach (9%) as well as cases where the results were different for each of the tools (6%). We conclude by discussing directions for future worksMestradoCiência da ComputaçãoMestre em Ciência da Computação2018/11129-8FAPES

Studying the Characteristics of AIOps Projects on GitHub

Artificial Intelligence for IT Operations (AIOps) leverages AI approaches to handle the massive data generated during the operations of software systems. Prior works have proposed various AIOps solutions to support different tasks in system operations and maintenance (e.g., anomaly detection). In this work, we investigate open-source AIOps projects in-depth to understand the characteristics of AIOps in practice. We first carefully identify a set of AIOps projects from GitHub and analyze their repository metrics (e.g., the used programming languages). Then, we qualitatively study the projects to understand their input data, analysis techniques, and goals. Finally, we analyze the quality of these projects using different quality metrics, such as the number of bugs. We also sample two sets of baseline projects from GitHub: a random sample of machine learning projects, and a random sample of general purpose projects. We compare different metrics of our identified AIOps projects with these baselines. Our results show a recent and growing interest in AIOps solutions. However, the quality metrics indicate that AIOps projects suffer from more issues than our baseline projects. We also pinpoint the most common issues in AIOps approaches and discuss the possible solutions to overcome them. Our findings help practitioners and researchers understand the current state of AIOps practices and sheds light to different ways to improve AIOps weak aspects. To the best of our knowledge, this work is the first to characterize open source AIOps projects.Comment: 31 pages, 6 pages of references, 8 figures, 12 table

Automatic assessment of Java code

[EN] Assessment is an integral part of education often used to evaluate students, but also to provide them with feedback. It is essential to ensure that assessment is fair, objective, and equally applied to all students. This holds, for instance, in multiple-choice tests, but, unfortunately, it is not ensured in the assessment of source code, which is still a manual and error-prone task. In this paper, we present JavAssess, a Java library with an API composed of around 200 methods to automatically inspect, test, mark, and correct Java code. It can be used to produce both black-box (based on output comparison) and white-box (based on the internal properties of the code) assessment tools. This means that it allows for marking the code even if it is only partially correct. We describe the library, how to use it, and we provide a complete example to automatically mark and correct a student's code. We also report the use of this system in a real university context to compare manual and automatic assessment in university courses. The study reports the average error in the marks produced by teachers when assessing source code manually, and it shows that the system automatically assesses around 50% of the work. (C) 2018 Elsevier Ltd. All rights reserved.This work has been partially supported by MINECO/AEI/FEDER (EU) under grants TIN2013-44742-C4-1-R and TIN2016-76843-C4-1-R, by the Generalitat Valenciana under grant PROMETEO-II/2015/013 (SmartLogic), and by the Universitat Politecnica de Valencia under grant PIME B18 and EICE HEGEA.Insa Cabrera, D.; Silva, J. (2018). Automatic assessment of Java code. Computer Languages Systems & Structures. 53:59-72. https://doi.org/10.1016/j.cl.2018.01.004S59725

You Cannot Fix What You Cannot Find! An Investigation of Fault Localization Bias in Benchmarking Automated Program Repair Systems

Properly benchmarking Automated Program Repair (APR) systems should contribute to the development and adoption of the research outputs by practitioners. To that end, the research community must ensure that it reaches significant milestones by reliably comparing state-of-the-art tools for a better understanding of their strengths and weaknesses. In this work, we identify and investigate a practical bias caused by the fault localization (FL) step in a repair pipeline. We propose to highlight the different fault localization configurations used in the literature, and their impact on APR systems when applied to the Defects4J benchmark. Then, we explore the performance variations that can be achieved by `tweaking' the FL step. Eventually, we expect to create a new momentum for (1) full disclosure of APR experimental procedures with respect to FL, (2) realistic expectations of repairing bugs in Defects4J, as well as (3) reliable performance comparison among the state-of-the-art APR systems, and against the baseline performance results of our thoroughly assessed kPAR repair tool. Our main findings include: (a) only a subset of Defects4J bugs can be currently localized by commonly-used FL techniques; (b) current practice of comparing state-of-the-art APR systems (i.e., counting the number of fixed bugs) is potentially misleading due to the bias of FL configurations; and (c) APR authors do not properly qualify their performance achievement with respect to the different tuning parameters implemented in APR systems.Comment: Accepted by ICST 201

EMPIRICAL ASSESSMENT OF THE IMPACT OF USING AUTOMATIC STATIC ANALYSIS ON CODE QUALITY

Automatic static analysis (ASA) tools analyze the source or compiled code looking for violations of recommended programming practices (called issues) that might cause faults or might degrade some dimensions of software quality. Antonio Vetro' has focused his PhD in studying how applying ASA impacts software quality, taking as reference point the different quality dimensions specified by the standard ISO/IEC 25010. The epistemological approach he used is that one of empirical software engineering. During his three years PhD, he's been conducting experiments and case studies on three main areas: Functionality/Reliability, Performance and Maintainability. He empirically proved that specific ASA issues had impact on these quality characteristics in the contexts under study: thus, removing them from the code resulted in a quality improvement. Vetro' has also investigated and proposed new research directions for this field: using ASA to improve software energy efficiency and to detect the problems deriving from the interaction of multiple languages. The contribution is enriched with the final recommendation of a generalized process for researchers and practitioners with a twofold goal: improve software quality through ASA and create a body of knowledge on the impact of using ASA on specific software quality dimensions, based on empirical evidence. This thesis represents a first step towards this goa