13 research outputs found
Beyond Accuracy: Assessing Software Documentation Quality
Good software documentation encourages good software engineering, but the
meaning of "good" documentation is vaguely defined in the software engineering
literature. To clarify this ambiguity, we draw on work from the data and
information quality community to propose a framework that decomposes
documentation quality into ten dimensions of structure, content, and style. To
demonstrate its application, we recruited technical editors to apply the
framework when evaluating examples from several genres of software
documentation. We summarise their assessments -- for example, reference
documentation and README files excel in quality whereas blog articles have more
problems -- and we describe our vision for reasoning about software
documentation quality and for the expansion and potential of a unified quality
framework.Comment: to appear in the Visions and Reflections Track of the ACM Joint
European Software Engineering Conference and Symposium on the Foundations of
Software Engineering 202
Speculative Analysis for Quality Assessment of Code Comments
Previous studies have shown that high-quality code comments assist developers
in program comprehension and maintenance tasks. However, the semi-structured
nature of comments, unclear conventions for writing good comments, and the lack
of quality assessment tools for all aspects of comments make their evaluation
and maintenance a non-trivial problem. To achieve high-quality comments, we
need a deeper understanding of code comment characteristics and the practices
developers follow. In this thesis, we approach the problem of assessing comment
quality from three different perspectives: what developers ask about commenting
practices, what they write in comments, and how researchers support them in
assessing comment quality.
Our preliminary findings show that developers embed various kinds of
information in class comments across programming languages. Still, they face
problems in locating relevant guidelines to write consistent and informative
comments, verifying the adherence of their comments to the guidelines, and
evaluating the overall state of comment quality. To help developers and
researchers in building comment quality assessment tools, we provide: (i) an
empirically validated taxonomy of comment convention-related questions from
various community forums, (ii) an empirically validated taxonomy of comment
information types from various programming languages, (iii) a
language-independent approach to automatically identify the information types,
and (iv) a comment quality taxonomy prepared from a systematic literature
review.Comment: 5 pages, 1 figure, conferenc
Deep Just-In-Time Inconsistency Detection Between Comments and Source Code
Natural language comments convey key aspects of source code such as
implementation, usage, and pre- and post-conditions. Failure to update comments
accordingly when the corresponding code is modified introduces inconsistencies,
which is known to lead to confusion and software bugs. In this paper, we aim to
detect whether a comment becomes inconsistent as a result of changes to the
corresponding body of code, in order to catch potential inconsistencies
just-in-time, i.e., before they are committed to a code base. To achieve this,
we develop a deep-learning approach that learns to correlate a comment with
code changes. By evaluating on a large corpus of comment/code pairs spanning
various comment types, we show that our model outperforms multiple baselines by
significant margins. For extrinsic evaluation, we show the usefulness of our
approach by combining it with a comment update model to build a more
comprehensive automatic comment maintenance system which can both detect and
resolve inconsistent comments based on code changes.Comment: Accepted in AAAI 202
Antipatterns in Software Classification Taxonomies
Empirical results in software engineering have long started to show that
findings are unlikely to be applicable to all software systems, or any domain:
results need to be evaluated in specified contexts, and limited to the type of
systems that they were extracted from. This is a known issue, and requires the
establishment of a classification of software types.
This paper makes two contributions: the first is to evaluate the quality of
the current software classifications landscape. The second is to perform a case
study showing how to create a classification of software types using a curated
set of software systems.
Our contributions show that existing, and very likely even new,
classification attempts are deemed to fail for one or more issues, that we
named as the `antipatterns' of software classification tasks. We collected 7 of
these antipatterns that emerge from both our case study, and the existing
classifications.
These antipatterns represent recurring issues in a classification, so we
discuss practical ways to help researchers avoid these pitfalls. It becomes
clear that classification attempts must also face the daunting task of
formulating a taxonomy of software types, with the objective of establishing a
hierarchy of categories in a classification.Comment: Accepted for publish at the Journal of Systems and Softwar
A Comprehensive Study of Governance Issues in Decentralized Finance Applications
Decentralized Finance (DeFi) is a prominent application of smart contracts,
representing a novel financial paradigm in contrast to centralized finance.
While DeFi applications are rapidly emerging on mainstream blockchain
platforms, their quality varies greatly, presenting numerous challenges,
particularly in terms of their governance mechanisms. In this paper, we present
a comprehensive study of governance issues in DeFi applications. Drawing upon
insights from industry reports and academic research articles, we develop a
taxonomy to categorize these governance issues. We collect and build a dataset
of 4,446 audit reports from 17 Web3 security companies, categorizing their
governance issues according to our constructed taxonomy. We conducted a
thorough analysis of governance issues and identified vulnerabilities in
governance design and implementation, e.g., voting sybil attack and proposal
front-running. Our findings highlight a significant observation: the disparity
between smart contract code and DeFi whitepapers plays a central role in these
governance issues. As an initial step to address the challenges of
code-whitepaper consistency checks for DeFi applications, we built a
machine-learning-based prototype, and validated its performance on eight widely
used DeFi projects, achieving a 56.14% F1 score and a 80% recall. Our study
culminates in providing several key practical implications for various DeFi
stakeholders, including developers, users, researchers, and regulators, aiming
to deepen the understanding of DeFi governance issues and contribute to the
robust growth of DeFi systems
Assessing Comment Quality in Object-Oriented Languages
Previous studies have shown that high-quality code comments support developers in software maintenance and program comprehension tasks. However, the semi-structured nature of comments, several conventions to write comments, and the lack of quality assessment tools for all aspects of comments make comment evaluation and maintenance a non-trivial problem. To understand the specification of high-quality comments to build effective assessment tools, our thesis emphasizes acquiring a multi-perspective view of the comments, which can be approached by analyzing (1) the academic support for comment quality assessment, (2) developer commenting practices across languages, and (3) developer concerns about comments.
Our findings regarding the academic support for assessing comment quality showed that researchers primarily focus on Java in the last decade even though the trend of using polyglot environments in software projects is increasing. Similarly, the trend of analyzing specific types of code comments (method comments, or inline comments) is increasing, but the studies rarely analyze class comments. We found 21 quality attributes that researchers consider to assess comment quality, and manual assessment is still the most commonly used technique to assess various quality attributes. Our analysis of developer commenting practices showed that developers embed a mixed level of details in class comments, ranging from high-level class overviews to low-level implementation details across programming languages. They follow style guidelines regarding what information to write in class comments but violate the structure and syntax guidelines. They primarily face problems locating relevant guidelines to write consistent and informative comments, verifying the adherence of their comments to the guidelines, and evaluating the overall state of comment quality.
To help researchers and developers in building comment quality assessment tools, we contribute: (i) a systematic literature review (SLR) of ten years (2010–2020) of research on assessing comment quality, (ii) a taxonomy of quality attributes used to assess comment quality, (iii) an empirically validated taxonomy of class comment information types from three programming languages, (iv) a multi-programming-language approach to automatically identify the comment information types, (v) an empirically validated taxonomy of comment convention-related questions and recommendation from various Q&A forums, and (vi) a tool to gather discussions from multiple developer sources, such as Stack Overflow, and mailing lists.
Our contributions provide various kinds of empirical evidence of the developer’s interest in reducing efforts in the software documentation process, of the limited support developers get in automatically assessing comment quality, and of the challenges they face in writing high-quality comments. This work lays the foundation for future effective comment quality assessment tools and techniques
A Decade of Code Comment Quality Assessment: A Systematic Literature Review
Code comments are important artifacts in software systems and play a
paramount role in many software engineering (SE) tasks related to maintenance
and program comprehension. However, while it is widely accepted that high
quality matters in code comments just as it matters in source code, assessing
comment quality in practice is still an open problem. First and foremost, there
is no unique definition of quality when it comes to evaluating code comments.
The few existing studies on this topic rather focus on specific attributes of
quality that can be easily quantified and measured. Existing techniques and
corresponding tools may also focus on comments bound to a specific programming
language, and may only deal with comments with specific scopes and clear goals
(e.g., Javadoc comments at the method level, or in-body comments describing
TODOs to be addressed). In this paper, we present a Systematic Literature
Review (SLR) of the last decade of research in SE to answer the following
research questions: (i) What types of comments do researchers focus on when
assessing comment quality? (ii) What quality attributes (QAs) do they consider?
(iii) Which tools and techniques do they use to assess comment quality?, and
(iv) How do they evaluate their studies on comment quality assessment in
general? Our evaluation, based on the analysis of 2353 papers and the actual
review of 47 relevant ones, shows that (i) most studies and techniques focus on
comments in Java code, thus may not be generalizable to other languages, and
(ii) the analyzed studies focus on four main QAs of a total of 21 QAs
identified in the literature, with a clear predominance of checking consistency
between comments and the code. We observe that researchers rely on manual
assessment and specific heuristics rather than the automated assessment of the
comment quality attributes
Assessing Comment Quality in Object-Oriented Languages
Previous studies have shown that high-quality code comments support developers in software maintenance and program comprehension tasks. However, the semi-structured nature of comments, several conventions to write comments, and the lack of quality assessment tools for all aspects of comments make comment evaluation and maintenance a non-trivial problem. To understand the specification of high-quality comments to build effective assessment tools, our thesis emphasizes acquiring a multi-perspective view of the comments, which can be approached by analyzing (1) the academic support for comment quality assessment, (2) developer commenting practices across languages, and (3) developer concerns about comments.
Our findings regarding the academic support for assessing comment quality showed that researchers primarily focus on Java in the last decade even though the trend of using polyglot environments in software projects is increasing. Similarly, the trend of analyzing specific types of code comments (method comments, or inline comments) is increasing, but the studies rarely analyze class comments. We found 21 quality attributes that researchers consider to assess comment quality, and manual assessment is still the most commonly used technique to assess various quality attributes. Our analysis of developer commenting practices showed that developers embed a mixed level of details in class comments, ranging from high-level class overviews to low-level implementation details across programming languages. They follow style guidelines regarding what information to write in class comments but violate the structure and syntax guidelines. They primarily face problems locating relevant guidelines to write consistent and informative comments, verifying the adherence of their comments to the guidelines, and evaluating the overall state of comment quality.
To help researchers and developers in building comment quality assessment tools, we contribute: (i) a systematic literature review (SLR) of ten years (2010–2020) of research on assessing comment quality, (ii) a taxonomy of quality attributes used to assess comment quality, (iii) an empirically validated taxonomy of class comment information types from three programming languages, (iv) a multi-programming-language approach to automatically identify the comment information types, (v) an empirically validated taxonomy of comment convention-related questions and recommendation from various Q&A forums, and (vi) a tool to gather discussions from multiple developer sources, such as Stack Overflow, and mailing lists.
Our contributions provide various kinds of empirical evidence of the developer’s interest in reducing efforts in the software documentation process, of the limited support developers get in automatically assessing comment quality, and of the challenges they face in writing high-quality comments. This work lays the foundation for future effective comment quality assessment tools and techniques