117 research outputs found
Large Language Models for Software Engineering: A Systematic Literature Review
Large Language Models (LLMs) have significantly impacted numerous domains,
notably including Software Engineering (SE). Nevertheless, a well-rounded
understanding of the application, effects, and possible limitations of LLMs
within SE is still in its early stages. To bridge this gap, our systematic
literature review takes a deep dive into the intersection of LLMs and SE, with
a particular focus on understanding how LLMs can be exploited in SE to optimize
processes and outcomes. Through a comprehensive review approach, we collect and
analyze a total of 229 research papers from 2017 to 2023 to answer four key
research questions (RQs). In RQ1, we categorize and provide a comparative
analysis of different LLMs that have been employed in SE tasks, laying out
their distinctive features and uses. For RQ2, we detail the methods involved in
data collection, preprocessing, and application in this realm, shedding light
on the critical role of robust, well-curated datasets for successful LLM
implementation. RQ3 allows us to examine the specific SE tasks where LLMs have
shown remarkable success, illuminating their practical contributions to the
field. Finally, RQ4 investigates the strategies employed to optimize and
evaluate the performance of LLMs in SE, as well as the common techniques
related to prompt optimization. Armed with insights drawn from addressing the
aforementioned RQs, we sketch a picture of the current state-of-the-art,
pinpointing trends, identifying gaps in existing research, and flagging
promising areas for future study
Evaluating the Likelihood of Bug Inducing Commits Using Metrics Trend Analysis
Continuous software engineering principles advocate a release-small, release-often process model, where new functionality is added to a system, in small increments and very frequently. In such a process model, every time a change is introduced it is important to identify as early as possible, whether the system has entered a state where faults are more likely to occur. In this paper, we present a method that is based on process, quality, and source code metrics to evaluate the likelihood that an imminent bug-inducing commit is highly probable. More specifically, the method analyzes the correlations and the rate of change of selected metrics. The findings from the technical debt dataset extracted data from SonarQube indicate that before bug-inducing commits, metrics that otherwise are not correlated, suddenly exhibit a high correlation or a high rate of change. This metric behavior can then be used for assessing an impending bug-inducing commit. The technique is programing language agnostic, based on metrics that can be extracted without the use of specialized parsers and can be applied to forewarn developers that a file, or a collection of files, has entered a state where faults are highly probable
Data Science Techniques for Modelling Execution Tracing Quality
This research presents how to handle a research problem when the research variables are still unknown, and
no quantitative study is possible; how to identify the research variables, to be able to perform a quantitative
research, how to collect data by means of the research variables identified, and how to carry out modelling
with the considerations of the specificities of the problem domain. In addition, validation is also encompassed
in the scope of modelling in the current study. Thus, the work presented in this thesis comprises the typical
stages a complex data science problem requires, including qualitative and quantitative research, data collection,
modelling of vagueness and uncertainty, and the leverage of artificial intelligence to gain such insights, which
are impossible with traditional methods.
The problem domain of the research conducted encompasses software product quality modelling, and assessment,
with particular focus on execution tracing quality. The terms execution tracing quality and logging are
used interchangeably throughout the thesis.
The research methods and mathematical tools used allow considering uncertainty and vagueness inherently
associated with the quality measurement and assessment process through which reality can be approximated
more appropriately in comparison to plain statistical modelling techniques. Furthermore, the modelling approach
offers direct insights into the problem domain by the application of linguistic rules, which is an additional
advantage.
The thesis reports (1) an in-depth investigation of all the identified software product quality models, (2) a
unified summary of the identified software product quality models with their terminologies and concepts, (3)
the identification of the variables influencing execution tracing quality, (4) the quality model constructed to
describe execution tracing quality, and (5) the link of the constructed quality model to the quality model of the
ISO/IEC 25010 standard, with the possibility of tailoring to specific project needs.
Further work, outside the frames of this PhD thesis, would also be useful as presented in the study: (1) to define
application-project profiles to assist tailoring the quality model for execution tracing to specific application and
project domains, and (2) to approximate the present quality model for execution tracing, within defined bounds,
by simpler mathematical approaches.
In conclusion, the research contributes to (1) supporting the daily work of software professionals, who need
to analyse execution traces; (2) raising awareness that execution tracing quality has a huge impact on software
development, software maintenance and on the professionals involved in the different stages of the software
development life-cycle; (3) providing a framework in which the present endeavours for log improvements
can be placed, and (4) suggesting an extension of the ISO/IEC 25010 standard by linking the constructed
quality model to that. In addition, in the scope of the qualitative research methodology, the current PhD thesis
contributes to the knowledge of research methods with determining a saturation point in the course of the data
collection process
Evolutionary Algorithms in Engineering Design Optimization
Evolutionary algorithms (EAs) are population-based global optimizers, which, due to their characteristics, have allowed us to solve, in a straightforward way, many real world optimization problems in the last three decades, particularly in engineering fields. Their main advantages are the following: they do not require any requisite to the objective/fitness evaluation function (continuity, derivability, convexity, etc.); they are not limited by the appearance of discrete and/or mixed variables or by the requirement of uncertainty quantification in the search. Moreover, they can deal with more than one objective function simultaneously through the use of evolutionary multi-objective optimization algorithms. This set of advantages, and the continuously increased computing capability of modern computers, has enhanced their application in research and industry. From the application point of view, in this Special Issue, all engineering fields are welcomed, such as aerospace and aeronautical, biomedical, civil, chemical and materials science, electronic and telecommunications, energy and electrical, manufacturing, logistics and transportation, mechanical, naval architecture, reliability, robotics, structural, etc. Within the EA field, the integration of innovative and improvement aspects in the algorithms for solving real world engineering design problems, in the abovementioned application fields, are welcomed and encouraged, such as the following: parallel EAs, surrogate modelling, hybridization with other optimization techniques, multi-objective and many-objective optimization, etc
Recommended from our members
Learning to Edit Code : Towards Building General Purpose Models for Source Code Editing
The way software developers edit code day-to-day tends to be repetitive, often using existing code elements. Many researchers have tried to automate the repetitive code editing process by mining specific change templates. However, such templates are often manually implemented for automated applications. Consequently, such template-based automated code editing is very tedious to implement. In addition, template-based code editing is often narrowly-scoped and low noise tolerant. Machine Learning, specially deep learning-based techniques, could help us solve these problems because of their generalization and noise tolerance capacities.
The advancement of deep neural networks and the availability of vast open-source evolutionary data opens up the possibility of automatically learning those templates from the wild and applying those in the appropriate context. However, deep neural network-based modeling for code changes, and code, in general, introduces some specific problems that need specific attention from the research community. For instance, source code exhibit strictly defined syntax and semantics inherited from the properties of Programming Language (PL). In addition, source code vocabulary (possible number of tokens) can be arbitrarily large.
This dissertation formulates the problem of automated code editing as a multi-modal translation problem, where, given a piece of code, the context, and some guidance, the objective is to generate edited code. In particular, we divide the problem into two sub-problems — source code understanding and generation. We empirically show that the deep neural networks (models in general) for these problems should be aware of the PL-properties (i.e., syntax, semantics). This dissertation investigates two primary directions of endowing the models with knowledge about PL-properties — (i) explicit encoding: where we design models catering to a specific property, and (ii) implicit encoding: where we train a very-large model to learn these properties from very large corpus of source code in unsupervised ways.
With implicit encoding, we custom design the model to cater to the need for that property. As an example of such models, we developed CODIT — a tree-based neural model for syntactic correctness. We design CODIT based on the Context Free Grammar of the programming language. Instead of generating source code, CODIT first generates the tree structure by sampling the production rule from the CFG. Such a mechanism prohibits infeasible production rule selection. In the later stage, CODIT generates the edited code conditioned on the tree generated earlier. Suchconditioning makes the edited code syntactically correct. CODIT showed promise in learning code edit patterns in the wild and effectiveness in automatic program repair. In another empirical study, we showed that a graph-based model is better suitable for source code understanding tasks such as vulnerability detection.
On the other hand, with implicit encoding, we use a very large (with several hundred million parameters) yet generic model. However, we pre-train these models on a super-large (usually hundreds of gigabytes) collection of source code and code metadata. We empirically show that if sufficiently pre-trained, such models are capable enough to learn PL properties such as syntax and semantics. In this dissertation, we developed two such pre-trained models, with two different learning objectives. First, we developed PLBART— the first-ever pre-trained encoder-decoder-based model for source code and show that such pre-train enables the model to generate syntactically and semantically correct code. Further, we show an in-depth empirical study on using PLBART in automated code editing. Finally, we develop another pre-trained model — NatGen to encode the natural coding convention followed by developers into the model. To design NatGen, we first deliberately modify the code from the developers’ written version preserving the original semantics. We call such transformations ‘de-naturalizing’ transformations. Following the previous studies on induced unnaturalness in code, we defined several such ‘de-naturalizing’ transformations and applied those to developer-written code. We pre-train NatGen to reverse the effect of these transformations. That way, NatGen learns to generate code similar to the developers’ written by undoing any unnaturalness induced by our forceful ‘de-naturalizing‘ transformations. NatGen has performed well in code editing and other source code generation tasks.
The models and empirical studies we performed while writing this dissertation go beyond the scope of automated code editing and are applicable to other software engineering automation problems such as Code translation, Code summarization, Code generation, Vulnerability detection,Clone detection, etc. Thus, we believe this dissertation will influence and contribute to the advancement of AI4SE and PLP
- …