Revisiting Sentiment Analysis for Software Engineering in the Era of Large Language Models

Software development is an inherently collaborative process, where various stakeholders frequently express their opinions and emotions across diverse platforms. Recognizing the sentiments conveyed in these interactions is crucial for the effective development and ongoing maintenance of software systems. Over the years, many tools have been proposed to aid in sentiment analysis, but accurately identifying the sentiments expressed in software engineering datasets remains challenging. Although fine-tuned smaller large language models (sLLMs) have shown potential in handling software engineering tasks, they struggle with the shortage of labeled data. With the emergence of bigger large language models (bLLMs), it is pertinent to investigate whether they can handle this challenge in the context of sentiment analysis for software engineering. In this work, we undertake a comprehensive empirical study using five established datasets. We assess the performance of three open-source bLLMs in both zero-shot and few-shot scenarios. Additionally, we compare them with fine-tuned sLLMs. Our experimental findings demonstrate that bLLMs exhibit state-of-the-art performance on datasets marked by limited training data and imbalanced distributions. bLLMs can also achieve excellent performance under a zero-shot setting. However, when ample training data is available or the dataset exhibits a more balanced distribution, fine-tuned sLLMs can still achieve superior results.Comment: Submitted to TOSE

Prompt Sapper: A LLM-Empowered Production Tool for Building AI Chains

The emergence of foundation models, such as large language models (LLMs) GPT-4 and text-to-image models DALL-E, has opened up numerous possibilities across various domains. People can now use natural language (i.e. prompts) to communicate with AI to perform tasks. While people can use foundation models through chatbots (e.g., ChatGPT), chat, regardless of the capabilities of the underlying models, is not a production tool for building reusable AI services. APIs like LangChain allow for LLM-based application development but require substantial programming knowledge, thus posing a barrier. To mitigate this, we propose the concept of AI chain and introduce the best principles and practices that have been accumulated in software engineering for decades into AI chain engineering, to systematise AI chain engineering methodology. We also develop a no-code integrated development environment, Prompt Sapper, which embodies these AI chain engineering principles and patterns naturally in the process of building AI chains, thereby improving the performance and quality of AI chains. With Prompt Sapper, AI chain engineers can compose prompt-based AI services on top of foundation models through chat-based requirement analysis and visual programming. Our user study evaluated and demonstrated the efficiency and correctness of Prompt Sapper.Comment: 23 pages, 5 figures, accepted to TOSEM 202

Performance Analysis of Legacy Perl Software via Batch and Interactive Trace Visualization

Performing an analysis of established software usually is challenging. Based on reverse engineering through dynamic analysis, it is possible to perform a software performance analysis, in order to detect performance bottlenecks or issues. This process is often divided into two consecutive tasks. The first task concerns the monitoring of the legacy software, and the second task covers analysing and visualizing the results. Dynamic analysis is usually addressed via trace visualization, but finding an appropriate representation for a specific issue still remains a great challenge. In this paper we report on our performance analysis of the Perl-based open repository software EPrints, which has now been continuously developed for more than fifteen years. We analyse and evaluate the software using the Kieker monitoring framework, and apply and combine two types of visualization tools, namely Graphviz and Gephi. More precisely, we employ Kieker to reconstruct architectural models from recorded monitoring data, based on dynamic analysis, and Graphviz respectively Gephi for further analysis and visualization of our monitoring results. We acquired knowledge of the software through our instrumentation and analysis via Kieker and the combined visualization of the two aforementioned tools. This allowed us, in collaboration with the EPrints development team, to reverse engineer their software EPrints, to give new and unexpected insights, and to detect potential bottlenecks

High frequency trading system design and process management

Thesis (S.M.)--Massachusetts Institute of Technology, System Design and Management Program, 2009.Cataloged from PDF version of thesis.Includes bibliographical references (p. 78-79).Trading firms nowadays are highly reliant on data mining, computer modeling and software development. Financial analysts perform many similar tasks to those in software and manufacturing industries. However, the finance industry has not yet fully adopted high-standard systems engineering frameworks and process management approaches that have been successful in the software and manufacturing industries. Many of the traditional methodologies for product design, quality control, systematic innovation, and continuous improvement found in engineering disciplines can be applied to the finance field. This thesis shows how the knowledge acquired from engineering disciplines can improve the design and processes management of high frequency trading systems. High frequency trading systems are computation-based. These systems are automatic or semi-automatic software systems that are inherently complex and require a high degree of design precision. The design of a high frequency trading system links multiple fields, including quantitative finance, system design and software engineering. In the finance industry, where mathematical theories and trading models are relatively well researched, the ability to implement these designs in real trading practices is one of the key elements of an investment firm's competitiveness. The capability of converting investment ideas into high performance trading systems effectively and efficiently can give an investment firm a huge competitive advantage.(cont.) This thesis provides a detailed study composed of high frequency trading system design, system modeling and principles, and processes management for system development. Particular emphasis is given to backtesting and optimization, which are considered the most important parts in building a trading system. This research builds system engineering models that guide the development process. It also uses experimental trading systems to verify and validate principles addressed in this thesis. Finally, this thesis concludes that systems engineering principles and frameworks can be the key to success for implementing high frequency trading or quantitative investment systems.by Xiangguang Xiao.S.M

Operational design co-ordination : an agent based approach

Operational design co-ordination has been identified as the basis for an approach to engineering design management that is more comprehensive than those that currently exist. As such, an integrated and holistic approach to operational design co-ordination has been developed that enables design to be managed in a coherent, appropriate and timely manner. Furthermore, the approach has been implemented within an agent-based software system, called the Design Co-ordination System, which has been applied to an industrial case study involving the computational design analysis of turbine blades. This application demonstrates that managing and adjusting in real-time in an operationally co-ordinated manner enables reductions in the time taken to complete the turbine blade design process to be achieved

Empirical analyses of the factors affecting confirmation bias and the effects of confirmation bias on software developer/tester performance

Background: During all levels of software testing, the goal should be to fail the code. However, software developers and testers are more likely to choose positive tests rather than negative ones due to the phenomenon called confirmation bias. Confirmation bias is defined as the tendency of people to verify their hypotheses rather than refuting them. In the literature, there are theories about the possible effects of confirmation bias on software development and testing. Due to the tendency towards positive tests, most of the software defects remain undetected, which in turn leads to an increase in software defect density. Aims: In this study, we analyze factors affecting confirmation bias in order to discover methods to circumvent confirmation bias. The factors, we investigate are experience in software development/testing and reasoning skills that can be gained through education. In addition, we analyze the effect of confirmation bias on software developer and tester performance. Method: In order to measure and quantify confirmation bias levels of software developers/testers, we prepared pen-and-paper and interactive tests based on two tasks from cognitive psychology literature. These tests were conducted on the 36 employees of a large scale telecommunication company in Europe as well as 28 graduate computer engineering students of Bogazici University, resulting in a total of 64 subjects. We evaluated the outcomes of these tests using the metrics we proposed in addition to some basic methods which we inherited from the cognitive psychology literature. Results: Results showed that regardless of experience in software development/testing, abilities such as logical reasoning and strategic hypotheses testing are differentiating factors in low confirmation bias levels. Moreover, the results of the analysis to investigate the relationship between code defect density and confirmation bias levels of software developers and testers showed that there is a direct correlation between confirmation bias and defect proneness of the code. Conclusions: Our findings show that having strong logical reasoning and hypothesis testing skills are differentiating factors in the software developer/tester performance in terms of defect rates. We recommend that companies should focus on improving logical reasoning and hypothesis testing skills of their employees by designing training programs. As future work, we plan to replicate this study in other software development companies. Moreover, we will use confirmation bias metrics in addition to product and process metrics in for software defect prediction. We believe that confirmation bias metrics would improve the prediction performance of learning based defect prediction models which we have been building over a decade

Search-Based Software Maintenance and Testing

2012 - 2013In software engineering there are many expensive tasks that are performed during development and maintenance activities. Therefore, there has been a lot of e ort to try to automate these tasks in order to signi cantly reduce the development and maintenance cost of software, since the automation would require less human resources. One of the most used way to make such an automation is the Search-Based Software Engineering (SBSE), which reformulates traditional software engineering tasks as search problems. In SBSE the set of all candidate solutions to the problem de nes the search space while a tness function di erentiates between candidate solutions providing a guidance to the optimization process. After the reformulation of software engineering tasks as optimization problems, search algorithms are used to solve them. Several search algorithms have been used in literature, such as genetic algorithms, genetic programming, simulated annealing, hill climbing (gradient descent), greedy algorithms, particle swarm and ant colony. This thesis investigates and proposes the usage of search based approaches to reduce the e ort of software maintenance and software testing with particular attention to four main activities: (i) program comprehension; (ii) defect prediction; (iii) test data generation and (iv) test suite optimiza- tion for regression testing. For program comprehension and defect prediction, this thesis provided their rst formulations as optimization problems and then proposed the usage of genetic algorithms to solve them. More precisely, this thesis investigates the peculiarity of source code against textual documents written in natural language and proposes the usage of Genetic Algorithms (GAs) in order to calibrate and assemble IR-techniques for di erent software engineering tasks. This thesis also investigates and proposes the usage of Multi-Objective Genetic Algorithms (MOGAs) in or- der to build multi-objective defect prediction models that allows to identify defect-prone software components by taking into account multiple and practical software engineering criteria. Test data generation and test suite optimization have been extensively investigated as search- based problems in literature . However, despite the huge body of works on search algorithms applied to software testing, both (i) automatic test data generation and (ii) test suite optimization present several limitations and not always produce satisfying results. The success of evolutionary software testing techniques in general, and GAs in particular, depends on several factors. One of these factors is the level of diversity among the individuals in the population, which directly a ects the exploration ability of the search. For example, evolutionary test case generation techniques that employ GAs could be severely a ected by genetic drift, i.e., a loss of diversity between solutions, which lead to a premature convergence of GAs towards some local optima. For these reasons, this thesis investigate the role played by diversity preserving mechanisms on the performance of GAs and proposed a novel diversity mechanism based on Singular Value Decomposition and linear algebra. Then, this mechanism has been integrated within the standard GAs and evaluated for evolutionary test data generation. It has been also integrated within MOGAs and empirically evaluated for regression testing. [edited by author]XII n.s

The uses of process modeling : a framework for understanding modeling formalisms

There is wide-spread recognition of the urgent need to improve software processes in order to improve the performance of software organizations. Process models are essential in achieving understanding and visibility of processes and are important for other uses including the analysis of processes for improvement. It has been increasingly difficult to compare and evaluate the variety of process modeling formalisms that have appeared in recent years without a clear understanding of precisely for what they will be used. The contribution of this paper is to provide an understanding and a fairly comprehensive catalog of the applications of process modeling for which formalisms may be used. The primary mechanism for doing this is a guided tour of the literature on process modeling supplemented by recent industrial experience. In the paper, basic definitions concerning processes, process descriptions and process modeling are reviewed and then uses of process modeling are surveyed under the following headings: communication among process participants, construction of new processes, control of processes, process· analysis, and process support by automation. Comments are offered on paradigms for process modeling formalisms and directions for future work to permit evolution of a discipline of process engineering are given