Toward the Automatic Classification of Self-Affirmed Refactoring

The concept of Self-Affirmed Refactoring (SAR) was introduced to explore how developers document their refactoring activities in commit messages, i.e., developers explicit documentation of refactoring operations intentionally introduced during a code change. In our previous study, we have manually identified refactoring patterns and defined three main common quality improvement categories including internal quality attributes, external quality attributes, and code smells, by only considering refactoring-related commits. However, this approach heavily depends on the manual inspection of commit messages. In this paper, we propose a two-step approach to first identify whether a commit describes developer-related refactoring events, then to classify it according to the refactoring common quality improvement categories. Specifically, we combine the N-Gram TF-IDF feature selection with binary and multiclass classifiers to build a new model to automate the classification of refactorings based on their quality improvement categories. We challenge our model using a total of 2,867 commit messages extracted from well engineered open-source Java projects. Our findings show that (1) our model is able to accurately classify SAR commits, outperforming the pattern-based and random classifier approaches, and allowing the discovery of 40 more relevent SAR patterns, and (2) our model reaches an F-measure of up to 90% even with a relatively small training datase

A Full-fledged Commit Message Quality Checker Based on Machine Learning

Commit messages (CMs) are an essential part of version control. By providing important context in regard to what has changed and why, they strongly support software maintenance and evolution. But writing good CMs is difficult and often neglected by developers. So far, there is no tool suitable for practice that automatically assesses how well a CM is written, including its meaning and context. Since this task is challenging, we ask the research question: how well can the CM quality, including semantics and context, be measured with machine learning methods? By considering all rules from the most popular CM quality guideline, creating datasets for those rules, and training and evaluating state-of-the-art machine learning models to check those rules, we can answer the research question with: sufficiently well for practice, with the lowest F$_1$ score of 82.9\%, for the most challenging task. We develop a full-fledged open-source framework that checks all these CM quality rules. It is useful for research, e.g., automatic CM generation, but most importantly for software practitioners to raise the quality of CMs and thus the maintainability and evolution speed of their software.Comment: published at COMPSAC'2

On the Documentation of Refactoring Types

Commit messages are the atomic level of software documentation. They provide a natural language description of the code change and its purpose. Messages are critical for software maintenance and program comprehension. Unlike documenting feature updates and bug fixes, little is known about how developers document their refactoring activities. Specifically, developers can perform multiple refactoring operations, including moving methods, extracting classes, renaming attributes, for various reasons, such as improving software quality, managing technical debt, and removing defects. Yet, there is no systematic study that analyzes the extent to which the documentation of refactoring accurately describes the refactoring operations performed at the source code level. Therefore, this paper challenges the ability of refactoring documentation, written in commit messages, to adequately predict the refactoring types, performed at the commit level. Our analysis relies on the text mining of commit messages to extract the corresponding features (i.e., keywords) that better represent each class (i.e., refactoring type). The extraction of text patterns, specific to each refactoring type (e.g., rename, extract, move, inline, etc.) allows the design of a model that verifies the consistency of these patterns with their corresponding refactoring. Such verification process can be achieved via automatically predicting, for a given commit, the method-level type of refactoring being applied, namely Extract Method, Inline Method, Move Method, Pull-up Method, Push-down Method, and Rename Method. We compared various classifiers, and a baseline keyword-based approach, in terms of their prediction performance, using a dataset of 5004 commits. Our main findings show that the complexity of refactoring type prediction varies from one type to another. Rename Method and Extract Method were found to be the best documented refactoring activities, while Pull-up Method, and Push-down Method were the hardest to be identified via textual descriptions. Such findings bring the attention of developers to the necessity of paying more attention to the documentation of these types

Remedying Security Concerns at an Internet Scale

The state of security across the Internet is poor, and it has been so since the advent of the modern Internet. While the research community has made tremendous progress over the years in learning how to design and build secure computer systems, network protocols, and algorithms, we are far from a world where we can truly trust the security of deployed Internet systems. In reality, we may never reach such a world. Security concerns continue to be identified at scale through-out the software ecosystem, with thousands of vulnerabilities discovered each year. Meanwhile, attacks have become ever more frequent and consequential.As Internet systems will continue to be inevitably affected by newly found security concerns, the research community must develop more effective ways to remedy these issues. To that end, in this dissertation, we conduct extensive empirical measurements to understand how remediation occurs in practice for Internet systems, and explore methods for spurring improved remediation behavior. This dissertation provides a treatment of the complete remediation life cycle, investigating the creation, dissemination, and deployment of remedies. We start by focusing on security patches that address vulnerabilities, and analyze at scale their creation process, characteristics of the resulting fixes, and how these impact vulnerability remediation. We then investigate and systematize how administrators of Internet systems deploy software updates which patch vulnerabilities across the many machines they manage on behalf of organizations. Finally, we conduct the first systematic exploration of Internet-scale outreach efforts to disseminate information about security concerns and their remedies to system administrators, with an aim of driving their remediation decisions. Our results show that such outreach campaigns can effectively galvanize positive reactions.Improving remediation, particularly at scale, is challenging, as the problem space exhibits many dimensions beyond traditional computer technical considerations, including human, social, organizational, economic, and policy facets. To make meaningful progress, this work uses a diversity of empirical methods, from software data mining to user studies to Internet-wide network measurements, to systematically collect and evaluate large-scale datasets. Ultimately, this dissertation establishes broad empirical grounding on security remediation in practice today, as well as new approaches for improved remediation at an Internet scale

Are Multi-language Design Smells Fault-prone? An Empirical Study

Nowadays, modern applications are developed using components written in different programming languages. These systems introduce several advantages. However, as the number of languages increases, so does the challenges related to the development and maintenance of these systems. In such situations, developers may introduce design smells (i.e., anti-patterns and code smells) which are symptoms of poor design and implementation choices. Design smells are defined as poor design and coding choices that can negatively impact the quality of a software program despite satisfying functional requirements. Studies on mono-language systems suggest that the presence of design smells affects code comprehension, thus making systems harder to maintain. However, these studies target only mono-language systems and do not consider the interaction between different programming languages. In this paper, we present an approach to detect multi-language design smells in the context of JNI systems. We then investigate the prevalence of those design smells. Specifically, we detect 15 design smells in 98 releases of nine open-source JNI projects. Our results show that the design smells are prevalent in the selected projects and persist throughout the releases of the systems. We observe that in the analyzed systems, 33.95% of the files involving communications between Java and C/C++ contains occurrences of multi-language design smells. Some kinds of smells are more prevalent than others, e.g., Unused Parameters, Too Much Scattering, Unused Method Declaration. Our results suggest that files with multi-language design smells can often be more associated with bugs than files without these smells, and that specific smells are more correlated to fault-proneness than others

Leveraging Evolutionary Changes for Software Process Quality

Real-world software applications must constantly evolve to remain relevant. This evolution occurs when developing new applications or adapting existing ones to meet new requirements, make corrections, or incorporate future functionality. Traditional methods of software quality control involve software quality models and continuous code inspection tools. These measures focus on directly assessing the quality of the software. However, there is a strong correlation and causation between the quality of the development process and the resulting software product. Therefore, improving the development process indirectly improves the software product, too. To achieve this, effective learning from past processes is necessary, often embraced through post mortem organizational learning. While qualitative evaluation of large artifacts is common, smaller quantitative changes captured by application lifecycle management are often overlooked. In addition to software metrics, these smaller changes can reveal complex phenomena related to project culture and management. Leveraging these changes can help detect and address such complex issues. Software evolution was previously measured by the size of changes, but the lack of consensus on a reliable and versatile quantification method prevents its use as a dependable metric. Different size classifications fail to reliably describe the nature of evolution. While application lifecycle management data is rich, identifying which artifacts can model detrimental managerial practices remains uncertain. Approaches such as simulation modeling, discrete events simulation, or Bayesian networks have only limited ability to exploit continuous-time process models of such phenomena. Even worse, the accessibility and mechanistic insight into such gray- or black-box models are typically very low. To address these challenges, we suggest leveraging objectively [...]Comment: Ph.D. Thesis without appended papers, 102 page

Open source software ecosystems quality analysis from data sources

Background: Open source software (OSS) and software ecosystems (SECOs) are two consolidated research areas in software engineering. The adoption of OSS by firms, governments, researchers and practitioners has been increasing rapidly in the last decades, and in consequence, they find themselves in a new kind of ecosystem composed by software communities,foundations, developers and partners, namely Open Source Software Ecosystem (OSSECO). In order to perform a systematic quality evaluation of a SECO, it is necessary to define certain types of concrete elements. This means that measures and evaluations should be described (e.g., through thresholds or expert judgment). The quality evaluation of an OSSECO may serve several purposes, for example: adopters of the products of the OSSECO may want to know about the liveliness of the OSSECO (e.g., recent updates); software developers may want to know about the activeness (e.g., how many collaborators are involved and how active they are); and the OSSECO community itself to know about the OSSECO health (e.g., evolving in the right direction). However, the current approaches for evaluating software quality (even those specific for open source software) do not cover all the aspects relevant in an OSSECO from an ecosystem perspective. Goal: The main goal of this PhD thesis is to support the OSSECO quality evaluation by designing a framework that supports the quality evaluation of OSSECOs. Methods: To accomplish this goal, we have used and approach based on design science methodology by Wieringa [1] and the characterization of software engineering proposed by M. Shaw [2], in order to produce a set of artefacts to contribute in thequality evaluation of OSSECOs and to learn about the effects of using these artefacts in practice. Results: We have conducted a systematic mapping to characterize OSSECOs and designed the QuESo framework (a framework to evaluate the OSSECO quality) composed by three artifacts: (i) QuESo-model, a quality model for OSSECOs; (ii) QuESoprocess, a process for conducting OSSECO quality evaluations using the QuESo-model; and (iii) QuESo-tool, a software component to support semi-automatic quality evaluation of OSSECOs. Furthermore, this framework has been validated with a case study on Eclipse. Conclusions: This thesis has contributed to increase the knowledge and understanding of OSSECOs, and to support the qualityevaluation of OSSECOs. [ntecedentes: el software de código abierto (OSS, por sus siglas en inglés) y los ecosistemas de software (SECOs, por sus siglas en inglés) son dos áreas de investigación consolidadas en ingeniería de software. La adopción de OSS por parte de empresas, gobiernos, investigadores y profesionales se ha incrementado rápidamente en las últimas décadas, y, en consecuencia, todos ellos hacen parte de un nuevo tipo de ecosistema formado por comunidades de software, fundaciones, desarrolladores y socios denominado ecosistema de software de código abierto. (OSSECO, por sus siglas en inglés)). Para realizar una evaluación sistemática de la calidad de un SECO, es necesario definir ciertos tipos de elementos concretos. Esto significa que tanto las métricas como las evaluaciones deben ser descritos (por ejemplo, a través de datos históricos o el conocimiento de expertos). La evaluación de la calidad de un OSSECO puede ser de utilidad desde diferentes perspectivas, por ejemplo: los que adoptan los productos del OSSECO pueden querer conocer la vitalidad del OSSECO (por ejemplo, el número de actualizaciones recientes); los desarrolladores de software pueden querer saber sobre la actividad del OSSECO (por ejemplo, cuántos colaboradores están involucrados y qué tan activos son); incluso la propia comunidad del OSSECO para conocer el estado de salud del OSSECO (por ejemplo, si está evolucionando en la dirección correcta). Sin embargo, los enfoques actuales para evaluar la calidad del software (incluso aquellos específicos para el software de código abierto) no cubren todos los aspectos relevantes en un OSSECO desde una perspectiva ecosistémica. Objetivo: El objetivo principal de esta tesis doctoral es apoyar la evaluación de la calidad de OSSECO mediante el diseño de un marco de trabajo que ayude a la evaluación de la calidad de un OSSECO. Métodos: Para lograr este objetivo, hemos utilizado un enfoque basado en la metodología design science propuesta por Wieringa [1]. Adicionalmente, nos hemos basado en la caracterización de la ingeniería de software propuesta por M. Shaw [2], con el fin de construir un conjunto de artefactos que contribuyan en la evaluación de la calidad de un OSSECO y para conocer los efectos del uso de estos artefactos en la práctica. Resultados: Hemos realizado un mapeo sistemático para caracterizar los OSSECOs y hemos diseñado el marco de trabajo denominado QuESo (es un marco de trabajo para evaluar la calidad de los OSSECOs). QuESo a su vez está compuesto por tres artefactos: (i) QuESo-model, un modelo de calidad para OSSECOs; (ii) QuESo-process, un proceso para llevar a cabo las evaluaciones de calidad de OSSECOs utilizando el modelo QuESo; y (iii) QuESo-tool, un conjunto de componentes de software que apoyan la evaluación de calidad de los OSSECOs de manera semiautomática. QuESo ha sido validado con un estudio de caso sobre Eclipse. Conclusiones: esta tesis ha contribuido a aumentar el conocimiento y la comprensión de los OSSECOs, y tambien ha apoyado la evaluación de la calidad de los OSSECO

Open source software ecosystems quality analysis from data sources

Background: Open source software (OSS) and software ecosystems (SECOs) are two consolidated research areas in software engineering. The adoption of OSS by firms, governments, researchers and practitioners has been increasing rapidly in the last decades, and in consequence, they find themselves in a new kind of ecosystem composed by software communities,foundations, developers and partners, namely Open Source Software Ecosystem (OSSECO). In order to perform a systematic quality evaluation of a SECO, it is necessary to define certain types of concrete elements. This means that measures and evaluations should be described (e.g., through thresholds or expert judgment). The quality evaluation of an OSSECO may serve several purposes, for example: adopters of the products of the OSSECO may want to know about the liveliness of the OSSECO (e.g., recent updates); software developers may want to know about the activeness (e.g., how many collaborators are involved and how active they are); and the OSSECO community itself to know about the OSSECO health (e.g., evolving in the right direction). However, the current approaches for evaluating software quality (even those specific for open source software) do not cover all the aspects relevant in an OSSECO from an ecosystem perspective. Goal: The main goal of this PhD thesis is to support the OSSECO quality evaluation by designing a framework that supports the quality evaluation of OSSECOs. Methods: To accomplish this goal, we have used and approach based on design science methodology by Wieringa [1] and the characterization of software engineering proposed by M. Shaw [2], in order to produce a set of artefacts to contribute in thequality evaluation of OSSECOs and to learn about the effects of using these artefacts in practice. Results: We have conducted a systematic mapping to characterize OSSECOs and designed the QuESo framework (a framework to evaluate the OSSECO quality) composed by three artifacts: (i) QuESo-model, a quality model for OSSECOs; (ii) QuESoprocess, a process for conducting OSSECO quality evaluations using the QuESo-model; and (iii) QuESo-tool, a software component to support semi-automatic quality evaluation of OSSECOs. Furthermore, this framework has been validated with a case study on Eclipse. Conclusions: This thesis has contributed to increase the knowledge and understanding of OSSECOs, and to support the qualityevaluation of OSSECOs. [ntecedentes: el software de código abierto (OSS, por sus siglas en inglés) y los ecosistemas de software (SECOs, por sus siglas en inglés) son dos áreas de investigación consolidadas en ingeniería de software. La adopción de OSS por parte de empresas, gobiernos, investigadores y profesionales se ha incrementado rápidamente en las últimas décadas, y, en consecuencia, todos ellos hacen parte de un nuevo tipo de ecosistema formado por comunidades de software, fundaciones, desarrolladores y socios denominado ecosistema de software de código abierto. (OSSECO, por sus siglas en inglés)). Para realizar una evaluación sistemática de la calidad de un SECO, es necesario definir ciertos tipos de elementos concretos. Esto significa que tanto las métricas como las evaluaciones deben ser descritos (por ejemplo, a través de datos históricos o el conocimiento de expertos). La evaluación de la calidad de un OSSECO puede ser de utilidad desde diferentes perspectivas, por ejemplo: los que adoptan los productos del OSSECO pueden querer conocer la vitalidad del OSSECO (por ejemplo, el número de actualizaciones recientes); los desarrolladores de software pueden querer saber sobre la actividad del OSSECO (por ejemplo, cuántos colaboradores están involucrados y qué tan activos son); incluso la propia comunidad del OSSECO para conocer el estado de salud del OSSECO (por ejemplo, si está evolucionando en la dirección correcta). Sin embargo, los enfoques actuales para evaluar la calidad del software (incluso aquellos específicos para el software de código abierto) no cubren todos los aspectos relevantes en un OSSECO desde una perspectiva ecosistémica. Objetivo: El objetivo principal de esta tesis doctoral es apoyar la evaluación de la calidad de OSSECO mediante el diseño de un marco de trabajo que ayude a la evaluación de la calidad de un OSSECO. Métodos: Para lograr este objetivo, hemos utilizado un enfoque basado en la metodología design science propuesta por Wieringa [1]. Adicionalmente, nos hemos basado en la caracterización de la ingeniería de software propuesta por M. Shaw [2], con el fin de construir un conjunto de artefactos que contribuyan en la evaluación de la calidad de un OSSECO y para conocer los efectos del uso de estos artefactos en la práctica. Resultados: Hemos realizado un mapeo sistemático para caracterizar los OSSECOs y hemos diseñado el marco de trabajo denominado QuESo (es un marco de trabajo para evaluar la calidad de los OSSECOs). QuESo a su vez está compuesto por tres artefactos: (i) QuESo-model, un modelo de calidad para OSSECOs; (ii) QuESo-process, un proceso para llevar a cabo las evaluaciones de calidad de OSSECOs utilizando el modelo QuESo; y (iii) QuESo-tool, un conjunto de componentes de software que apoyan la evaluación de calidad de los OSSECOs de manera semiautomática. QuESo ha sido validado con un estudio de caso sobre Eclipse. Conclusiones: esta tesis ha contribuido a aumentar el conocimiento y la comprensión de los OSSECOs, y tambien ha apoyado la evaluación de la calidad de los OSSECOsPostprint (published version

Extending the Reach of Fault Localization to Assist in Automated Debugging

Software debugging is one of the most time-consuming tasks in modern software maintenance. To assist developers with debugging, researchers have proposed fault localization techniques. These techniques aim to automate the process of locating faults in software, which can greatly reduce debugging time and assist developers in understanding the faults. Effective fault localization is also crucial for automated program repair techniques, as it helps identify potential faulty locations for patching. Despite recent efforts to advance fault localization techniques, their effectiveness is still limited. With the increasing complexity of modern software, fault localization may not always provide direct identification of the root causes of faults. Further, there is a lack of studies on their application in modern software development. Most prior studies have evaluated these techniques in traditional software development settings, where only a single snapshot of the system is considered. However, modern software development often involves continuous and fine-grained changes to the system. This dissertation proposes a series of approaches to explore new automated debugging solutions that can enhance software quality assurance and reliability practices, with a specific focus on extending the reach of fault localization in modern software development. The dissertation begins with an empirical study on user-reported logs in bug reports, revealing that re-constructed execution paths from these logs provide valuable debugging hints. To further assist developers in debugging, we propose using static analysis techniques for information-retrieval and path-guided fault localization. By leveraging execution paths from logs in bug reports, we can improve the effectiveness of fault localization techniques. Second, we investigate the characteristics of operational data in continuous integration that can help capture faults early in the testing phase. As there is currently no available continuous integration benchmark that incorporates continuous test execution and failure, we present T-Evos, a dataset that comprises various operational data in continuous integration settings. We propose automated fault localization techniques that integrate change information from continuous integration settings, and demonstrate that leveraging such fine-grained change information can significantly improve their effectiveness. Finally, the dissertation investigates the data cleanness in fault localization by examining developers' knowledge in fault-triggering tests. The study reveals a significant degradation in the performance of fault localization techniques when evaluated on faults without developer knowledge. Through case studies and experiments, the proposed techniques in this dissertation significantly improve the effectiveness of fault localization and facilitate their adoption in modern software development. Additionally, this dissertation provides valuable insights into new debugging solutions for future research